[jira] [Commented] (NIFI-5333) Create GetMongoRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640570#comment-16640570 ] ASF GitHub Bot commented on NIFI-5333: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3011 @MikeThomsen Will try to do by this weekend. > Create GetMongoRecord processor > --- > > Key: NIFI-5333 > URL: https://issues.apache.org/jira/browse/NIFI-5333 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > A processor similar to GetMongo that uses the record API should be created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3011: NIFI-5333 Added GetMongoRecord.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3011 @MikeThomsen Will try to do by this weekend. ---
[GitHub] nifi issue #3025: NIFI-5605 Added UpdateCassandra.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3025 I think this can go in with `PutCassandraQL`. `PutCassandraRecord` actually performs inserts only. ---
[jira] [Commented] (NIFI-5605) Create Cassandra update/delete processor
[ https://issues.apache.org/jira/browse/NIFI-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640569#comment-16640569 ] ASF GitHub Bot commented on NIFI-5605: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3025 I think this can go in with `PutCassandraQL`. `PutCassandraRecord` actually performs inserts only. > Create Cassandra update/delete processor > > > Key: NIFI-5605 > URL: https://issues.apache.org/jira/browse/NIFI-5605 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > There should be a processor that can update and delete from Cassandra tables. > Where QueryCassandra is geared toward reading data, this should be geared > toward data mutations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi pull request #137: MINIFI-471 Update to a more recent version of...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi/pull/137 ---
[jira] [Commented] (NIFI-5537) Create Neo4J cypher execution processor
[ https://issues.apache.org/jira/browse/NIFI-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640415#comment-16640415 ] ASF GitHub Bot commented on NIFI-5537: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2956 @mans2singh before we continue, do you mind joining in [this discussion](http://apache-nifi-developer-list.39713.n7.nabble.com/Graph-database-support-w-NiFi-tp18680p19706.html) that I reopened on the mailing list? > Create Neo4J cypher execution processor > --- > > Key: NIFI-5537 > URL: https://issues.apache.org/jira/browse/NIFI-5537 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.7.1 > Environment: All >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: graph, neo4j, node, relationship > > Create Nifi Neo4J cypher queries processor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2956 @mans2singh before we continue, do you mind joining in [this discussion](http://apache-nifi-developer-list.39713.n7.nabble.com/Graph-database-support-w-NiFi-tp18680p19706.html) that I reopened on the mailing list? ---
[jira] [Commented] (MINIFICPP-635) Typo in readme
[ https://issues.apache.org/jira/browse/MINIFICPP-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640411#comment-16640411 ] ASF GitHub Bot commented on MINIFICPP-635: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/413 MINIFICPP-635: Resolve typos Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-635 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/413.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #413 commit 590253f71a02e7e4a0447d2ce6104d63ff2631a6 Author: Marc Parisi Date: 2018-10-05T22:13:21Z MINIFICPP-635: Resolve typos > Typo in readme > -- > > Key: MINIFICPP-635 > URL: https://issues.apache.org/jira/browse/MINIFICPP-635 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #413: MINIFICPP-635: Resolve typos
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/413 MINIFICPP-635: Resolve typos Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-635 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/413.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #413 commit 590253f71a02e7e4a0447d2ce6104d63ff2631a6 Author: Marc Parisi Date: 2018-10-05T22:13:21Z MINIFICPP-635: Resolve typos ---
[jira] [Created] (MINIFICPP-635) Typo in readme
Mr TheSegfault created MINIFICPP-635: Summary: Typo in readme Key: MINIFICPP-635 URL: https://issues.apache.org/jira/browse/MINIFICPP-635 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Mr TheSegfault Assignee: Mr TheSegfault -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5664) LookupRecord cannot handle JSON array properties from MongoDBLookupService
[
https://issues.apache.org/jira/browse/NIFI-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640358#comment-16640358
]
ASF GitHub Bot commented on NIFI-5664:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3049#discussion_r223141579
--- Diff:
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
---
@@ -339,6 +339,11 @@ public static boolean isRecordTypeCompatible(final
Object value) {
return dest;
}
--- End diff --
Why wouldn't this be any List implementation?
> LookupRecord cannot handle JSON array properties from MongoDBLookupService
> --
>
> Key: NIFI-5664
> URL: https://issues.apache.org/jira/browse/NIFI-5664
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0, 1.7.1
>Reporter: Carl Gieringer
>Priority: Major
>
> We are using a LookupRecord processor to add an array property to a Record.
> The LookupRecord's `Lookup Service` is a MongoDBLookupService. The idea is
> for the LookupRecord to query Mongo documents by the Record's `_id`, to
> extract the array property at `f` on that document, and to insert that array
> into the Record at the RecordPath `f`. This flow results in the conversion
> error copied below, which completely prevents the FlowFile from proceeding
> (routes to Failure/Unmatched.) It appears that the `MongoDBLookupService` (or
> perhaps its `MongoDBControllerService`) stores Mongo document arrays as
> `ArrayList`s, which `DataTypeUtils#toArray` does not support. A simple fix
> appears to be to add a few lines to `DataTypeUtils#toArray` to support
> `ArrayList`s. I will provide a PR for this issue.
> This issue appears in 1.7.1, and is extant in 1.8.0-SNAPSHOT (2018-10-02
> b4c8e0179). I am not aware of any workaround to enrich a Record with an array
> from Mongo.
> LookupRecord Configuration
> ||Property||Value||
> |Result RecordPath|/f|
> |Record Result Contents|Insert Entire Record|
> |_id|/_id|
> MongoDBLookupService Configuration
> ||Property||Value||
> |Lookup Value Field|f|
> Conversion error (with irrelevant Mongo document and schema properties
> elided):
> {code:java}
> 2018-10-04 15:00:31,348 ERROR [Timer-Driven Process Thread-1]
> o.a.n.processors.standard.LookupRecord
> LookupRecord[id=339bf72c-91c3-36f0-ac14-bc8b734e2582] Failed to write
> MapRecord[{... f=[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}],
> i=[Ljava.lang.Object;@35dee8fc, ...] with schema {
> "namespace": "nifi",
> "name": "MongoPosting",
> "type": "record",
> "fields": [
> ...
> { "name": "f", "type": [
> "null",
> { "type": "array", "items": {
> "type": "record", "name": "PostingFeatureDetails", "fields": [
> { "name": "fid", "type": "string" },
> { "name": "l" , "type": ["null", "double"] }
> ]}
> }
> ]},
> ...
> ]
> } as a JSON Object due to
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
> java.util.ArrayList to Object Array for field f:
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
> java.util.ArrayList to Object Array for field f
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
> java.util.ArrayList to Object Array for field f
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.toArray(DataTypeUtils.java:347)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:153)
> at
>
[GitHub] nifi pull request #3049: NIFI-5664 Support ArrayList in DataTypeUtils#toArra...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3049#discussion_r223141579
--- Diff:
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java
---
@@ -339,6 +339,11 @@ public static boolean isRecordTypeCompatible(final
Object value) {
return dest;
}
--- End diff --
Why wouldn't this be any List implementation?
---
[GitHub] nifi-site pull request #30: Remove licensing comments as it is rendered in m...
GitHub user apiri opened a pull request: https://github.com/apache/nifi-site/pull/30 Remove licensing comments as it is rendered in markdown. Remove licensing comments as it is rendered in markdown. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apiri/nifi-site adjust-minifi-admin-guide Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-site/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit e39e12ea7328456960528ca8f9e7e264d0df1d23 Author: Aldrin Piri Date: 2018-10-05T21:09:18Z Remove licensing comments as it is rendered in markdown. ---
[jira] [Commented] (NIFI-5661) Add UI support for load balancing connections
[
https://issues.apache.org/jira/browse/NIFI-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640255#comment-16640255
]
ASF GitHub Bot commented on NIFI-5661:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3046#discussion_r223115867
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-connection.js
---
@@ -1341,6 +1363,52 @@
}
});
+// determine whether or not to show the load-balance
icon
+
connectionLabelContainer.select('text.load-balance-icon')
+.classed('hidden', function () {
+if (d.permissions.canRead) {
+return
!isLoadBalanceConfigured(d.component);
+} else {
+return true;
+}
+}).classed('load-balance-icon-active
fa-rotate-90', function (d) {
+return d.permissions.canRead &&
d.component.loadBalanceStatus === 'LOAD_BALANCE_ACTIVE';
+
+}).classed('load-balance-icon-184', function() {
+return isExpirationConfigured(d.component);
+
+}).classed('load-balance-icon-200', function() {
+return !isExpirationConfigured(d.component);
+
+}).attr('x', function() {
+return isExpirationConfigured(d.component) ?
184 : 200;
--- End diff --
When the user does not have permission to the connection, the `component`
here will not be set. We need to check this and handle accordingly anywhere we
are invoking `isExpirationConfigured` (see the above 3 cases) since it requires
the `component`.
> Add UI support for load balancing connections
> -
>
> Key: NIFI-5661
> URL: https://issues.apache.org/jira/browse/NIFI-5661
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> NIFI-5516 adds FlowFile load-balancing feature at connections. We need UI
> controls to:
> - Configure if a connection is load balanced and what strategy to use.
> - Indicator that load balancing is in progress
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5661) Add UI support for load balancing connections
[
https://issues.apache.org/jira/browse/NIFI-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640254#comment-16640254
]
ASF GitHub Bot commented on NIFI-5661:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3046#discussion_r223116414
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/css/connection-configuration.css
---
@@ -122,4 +122,16 @@ div.undefined {
line-height: 32px;
height: 32px;
padding: 0px 10px;
-}
\ No newline at end of file
+}
+
+.multi-column-settings {
--- End diff --
I don't think we need to duplicate this styles here. The styles from the
`connection-details.css` should be in scope.
> Add UI support for load balancing connections
> -
>
> Key: NIFI-5661
> URL: https://issues.apache.org/jira/browse/NIFI-5661
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> NIFI-5516 adds FlowFile load-balancing feature at connections. We need UI
> controls to:
> - Configure if a connection is load balanced and what strategy to use.
> - Indicator that load balancing is in progress
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3046: NIFI-5661: Adding Load Balance config UI
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3046#discussion_r223116414
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/css/connection-configuration.css
---
@@ -122,4 +122,16 @@ div.undefined {
line-height: 32px;
height: 32px;
padding: 0px 10px;
-}
\ No newline at end of file
+}
+
+.multi-column-settings {
--- End diff --
I don't think we need to duplicate this styles here. The styles from the
`connection-details.css` should be in scope.
---
[GitHub] nifi pull request #3046: NIFI-5661: Adding Load Balance config UI
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3046#discussion_r223115867
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-connection.js
---
@@ -1341,6 +1363,52 @@
}
});
+// determine whether or not to show the load-balance
icon
+
connectionLabelContainer.select('text.load-balance-icon')
+.classed('hidden', function () {
+if (d.permissions.canRead) {
+return
!isLoadBalanceConfigured(d.component);
+} else {
+return true;
+}
+}).classed('load-balance-icon-active
fa-rotate-90', function (d) {
+return d.permissions.canRead &&
d.component.loadBalanceStatus === 'LOAD_BALANCE_ACTIVE';
+
+}).classed('load-balance-icon-184', function() {
+return isExpirationConfigured(d.component);
+
+}).classed('load-balance-icon-200', function() {
+return !isExpirationConfigured(d.component);
+
+}).attr('x', function() {
+return isExpirationConfigured(d.component) ?
184 : 200;
--- End diff --
When the user does not have permission to the connection, the `component`
here will not be set. We need to check this and handle accordingly anywhere we
are invoking `isExpirationConfigured` (see the above 3 cases) since it requires
the `component`.
---
[GitHub] nifi-minifi pull request #137: MINIFI-471 Update to a more recent version of...
GitHub user apiri opened a pull request: https://github.com/apache/nifi-minifi/pull/137 MINIFI-471 Update to a more recent version of Jetty aligned with NiFi. Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apiri/nifi-minifi MINIFI-471 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #137 commit f2ce176d0b8b8bc54d8592d7f4ff2cd964cd1096 Author: Aldrin Piri Date: 2018-10-05T19:02:09Z MINIFI-471 Update to a more recent version of Jetty aligned with NiFi. ---
[jira] [Commented] (NIFI-5664) LookupRecord cannot handle JSON array properties from MongoDBLookupService
[
https://issues.apache.org/jira/browse/NIFI-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640124#comment-16640124
]
ASF GitHub Bot commented on NIFI-5664:
--
Github user snagacarl commented on the issue:
https://github.com/apache/nifi/pull/3049
Apache issue: https://issues.apache.org/jira/browse/NIFI-5664
> LookupRecord cannot handle JSON array properties from MongoDBLookupService
> --
>
> Key: NIFI-5664
> URL: https://issues.apache.org/jira/browse/NIFI-5664
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0, 1.7.1
>Reporter: Carl Gieringer
>Priority: Major
>
> We are using a LookupRecord processor to add an array property to a Record.
> The LookupRecord's `Lookup Service` is a MongoDBLookupService. The idea is
> for the LookupRecord to query Mongo documents by the Record's `_id`, to
> extract the array property at `f` on that document, and to insert that array
> into the Record at the RecordPath `f`. This flow results in the conversion
> error copied below, which completely prevents the FlowFile from proceeding
> (routes to Failure/Unmatched.) It appears that the `MongoDBLookupService` (or
> perhaps its `MongoDBControllerService`) stores Mongo document arrays as
> `ArrayList`s, which `DataTypeUtils#toArray` does not support. A simple fix
> appears to be to add a few lines to `DataTypeUtils#toArray` to support
> `ArrayList`s. I will provide a PR for this issue.
> This issue appears in 1.7.1, and is extant in 1.8.0-SNAPSHOT (2018-10-02
> b4c8e0179). I am not aware of any workaround to enrich a Record with an array
> from Mongo.
> LookupRecord Configuration
> ||Property||Value||
> |Result RecordPath|/f|
> |Record Result Contents|Insert Entire Record|
> |_id|/_id|
> MongoDBLookupService Configuration
> ||Property||Value||
> |Lookup Value Field|f|
> Conversion error (with irrelevant Mongo document and schema properties
> elided):
> {code:java}
> 2018-10-04 15:00:31,348 ERROR [Timer-Driven Process Thread-1]
> o.a.n.processors.standard.LookupRecord
> LookupRecord[id=339bf72c-91c3-36f0-ac14-bc8b734e2582] Failed to write
> MapRecord[{... f=[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}],
> i=[Ljava.lang.Object;@35dee8fc, ...] with schema {
> "namespace": "nifi",
> "name": "MongoPosting",
> "type": "record",
> "fields": [
> ...
> { "name": "f", "type": [
> "null",
> { "type": "array", "items": {
> "type": "record", "name": "PostingFeatureDetails", "fields": [
> { "name": "fid", "type": "string" },
> { "name": "l" , "type": ["null", "double"] }
> ]}
> }
> ]},
> ...
> ]
> } as a JSON Object due to
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
> java.util.ArrayList to Object Array for field f:
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
> java.util.ArrayList to Object Array for field f
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
> java.util.ArrayList to Object Array for field f
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.toArray(DataTypeUtils.java:347)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:153)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:115)
> at
> org.apache.nifi.json.WriteJsonResult.writeValue(WriteJsonResult.java:284)
> at
> org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:187)
> at
>
[GitHub] nifi issue #3049: NIFI-5664 Support ArrayList in DataTypeUtils#toArray
Github user snagacarl commented on the issue: https://github.com/apache/nifi/pull/3049 Apache issue: https://issues.apache.org/jira/browse/NIFI-5664 ---
[jira] [Commented] (NIFI-5664) LookupRecord cannot handle JSON array properties from MongoDBLookupService
[
https://issues.apache.org/jira/browse/NIFI-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640121#comment-16640121
]
ASF GitHub Bot commented on NIFI-5664:
--
GitHub user snagacarl opened a pull request:
https://github.com/apache/nifi/pull/3049
NIFI-5664 Support ArrayList in DataTypeUtils#toArray
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [Y] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [Y] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [Y] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [Y] Is your initial contribution a single, squashed commit?
### For code changes:
- [Y] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [Y] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [N/A] If applicable, have you updated the LICENSE file, including the
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition
to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Snagajob/nifi NIFI-5664
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3049.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3049
commit 30d0410f8f35cff772cc74fc36e4368d33310aec
Author: Carl Gieringer
Date: 2018-10-04T16:50:08Z
NIFI-5664 Support ArrayList in DataTypeUtils#toArray
> LookupRecord cannot handle JSON array properties from MongoDBLookupService
> --
>
> Key: NIFI-5664
> URL: https://issues.apache.org/jira/browse/NIFI-5664
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0, 1.7.1
>Reporter: Carl Gieringer
>Priority: Major
>
> We are using a LookupRecord processor to add an array property to a Record.
> The LookupRecord's `Lookup Service` is a MongoDBLookupService. The idea is
> for the LookupRecord to query Mongo documents by the Record's `_id`, to
> extract the array property at `f` on that document, and to insert that array
> into the Record at the RecordPath `f`. This flow results in the conversion
> error copied below, which completely prevents the FlowFile from proceeding
> (routes to Failure/Unmatched.) It appears that the `MongoDBLookupService` (or
> perhaps its `MongoDBControllerService`) stores Mongo document arrays as
> `ArrayList`s, which `DataTypeUtils#toArray` does not support. A simple fix
> appears to be to add a few lines to `DataTypeUtils#toArray` to support
> `ArrayList`s. I will provide a PR for this issue.
> This issue appears in 1.7.1, and is extant in 1.8.0-SNAPSHOT (2018-10-02
> b4c8e0179). I am not aware of any workaround to enrich a Record with an array
> from Mongo.
> LookupRecord Configuration
> ||Property||Value||
> |Result RecordPath|/f|
> |Record Result Contents|Insert Entire Record|
> |_id|/_id|
> MongoDBLookupService Configuration
> ||Property||Value||
> |Lookup Value Field|f|
> Conversion error (with irrelevant Mongo document and schema properties
> elided):
> {code:java}
> 2018-10-04 15:00:31,348 ERROR [Timer-Driven Process Thread-1]
> o.a.n.processors.standard.LookupRecord
> LookupRecord[id=339bf72c-91c3-36f0-ac14-bc8b734e2582] Failed to write
> MapRecord[{... f=[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
> Document{{fid=5762adf32fc74d0418104201, l=2.12}},
> Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
> Document{{fid=587542dd638e870838a79d08, l=1.0}},
> Document{{fid=5762addbffa7cf0980267acd,
[GitHub] nifi pull request #3049: NIFI-5664 Support ArrayList in DataTypeUtils#toArra...
GitHub user snagacarl opened a pull request: https://github.com/apache/nifi/pull/3049 NIFI-5664 Support ArrayList in DataTypeUtils#toArray Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [Y] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [Y] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [Y] Has your PR been rebased against the latest commit within the target branch (typically master)? - [Y] Is your initial contribution a single, squashed commit? ### For code changes: - [Y] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [Y] Have you written or updated unit tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [N/A] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [N/A] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [N/A] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Snagajob/nifi NIFI-5664 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3049.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3049 commit 30d0410f8f35cff772cc74fc36e4368d33310aec Author: Carl Gieringer Date: 2018-10-04T16:50:08Z NIFI-5664 Support ArrayList in DataTypeUtils#toArray ---
[jira] [Commented] (NIFI-3993) Upgrade embedded ZooKeeper version
[ https://issues.apache.org/jira/browse/NIFI-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640054#comment-16640054 ] Joseph Witt commented on NIFI-3993: --- definitely should not close this ticket. What moser wrote is what we should definitely test and do if feasible in my opinion > Upgrade embedded ZooKeeper version > -- > > Key: NIFI-3993 > URL: https://issues.apache.org/jira/browse/NIFI-3993 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.2.0 >Reporter: Mark Bean >Priority: Major > > In a Cluster configuration, Nodes are periodically disconnected from the > Cluster, and then reconnected. These events correspond to the following error: > ERROR [CommitProcessor:1] o.apache.zookeeper.server.NIOServerCnxn Unexpected > Exception: > java.nio.channels.CancelledKeyException: null > at sun.nio.ch.SelectionKeyImpl.ensureValid(SectionKeyImpl.java:73) > at sun.nio.ch.SelectionKeyImpl.interestOps(SelctionKeyImpl.java:77) > at > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) > at > org.apache.zookeeper.server.NIOServerCnXn.sendResopnse(NIOServerCnxn.java:1081) > at > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) > at > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) > This error was reported in ZooKeeper JIRA [1], and reported as fixed in > version 3.4.10, the current stable build. As additional confirmation, when > using a stand-alone ZK, 3.4.10, rather than the embedded ZK, the above error > was no longer observed. > Update NiFi to use ZK 3.4.10 > [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2044 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3993) Upgrade embedded ZooKeeper version
[ https://issues.apache.org/jira/browse/NIFI-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639942#comment-16639942 ] Mark Bean commented on NIFI-3993: - As noted, there are CVEs affecting 3.4.6, most notably "CVE-2018-8012: Apache ZooKeeper Quorum Peer mutual authentication". If for no other reason, I think this is enough justification to keep this ticket open. What specific services/components depend on ZK 3.4.6? Do they break if using ZK 3.4.10? > Upgrade embedded ZooKeeper version > -- > > Key: NIFI-3993 > URL: https://issues.apache.org/jira/browse/NIFI-3993 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.2.0 >Reporter: Mark Bean >Priority: Major > > In a Cluster configuration, Nodes are periodically disconnected from the > Cluster, and then reconnected. These events correspond to the following error: > ERROR [CommitProcessor:1] o.apache.zookeeper.server.NIOServerCnxn Unexpected > Exception: > java.nio.channels.CancelledKeyException: null > at sun.nio.ch.SelectionKeyImpl.ensureValid(SectionKeyImpl.java:73) > at sun.nio.ch.SelectionKeyImpl.interestOps(SelctionKeyImpl.java:77) > at > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) > at > org.apache.zookeeper.server.NIOServerCnXn.sendResopnse(NIOServerCnxn.java:1081) > at > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) > at > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) > This error was reported in ZooKeeper JIRA [1], and reported as fixed in > version 3.4.10, the current stable build. As additional confirmation, when > using a stand-alone ZK, 3.4.10, rather than the embedded ZK, the above error > was no longer observed. > Update NiFi to use ZK 3.4.10 > [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2044 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640012#comment-16640012
]
ASF GitHub Bot commented on NIFI-5663:
--
GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/3048
NIFI-5663: Ensure that when sort Node Identifiers that we use both th…
…e node's API Address as well as API Port, in case 2 nodes are running on
same host. Also ensure that when Local Node ID is determined that we update all
Load Balancing Partitions, if necessary
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi fix-NIFI-5663
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3048.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3048
commit 46397c16d7dd8c402fe7e7fc301893c27a924c73
Author: Mark Payne
Date: 2018-10-05T16:06:39Z
NIFI-5663: Ensure that when sort Node Identifiers that we use both the
node's API Address as well as API Port, in case 2 nodes are running on same
host. Also ensure that when Local Node ID is determined that we update all Load
Balancing Partitions, if necessary
> FlowFile load balancing keeps re-partitioning
> -
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Koji Kawamura
>Assignee: Mark Payne
>Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node (nifi0). Using existing
> multi-node clusters don't reproduce the issue.
> # Create GenerateFlowFile -> LogAttribute
> # Then set 'Partition by attribute' LB strategy at the connection
> # Add 2nd node, nifi1
> # Generate some FlowFiles. Then load-balance activity never finishes.
> With a 2-node cluster, for some reason, each node ended up having different
> queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug
> logs, I found each node has followings:
> * nifi0
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi1.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> * nifi1
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi0.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
> received FlowFiles between each other in case the calculated attribute value
> hash points to queuePartitions[0]. Following log is written endlessly:
> {code:java}
> 2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
>
[GitHub] nifi pull request #3048: NIFI-5663: Ensure that when sort Node Identifiers t...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3048 NIFI-5663: Ensure that when sort Node Identifiers that we use both th⦠â¦e node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi fix-NIFI-5663 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3048.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3048 commit 46397c16d7dd8c402fe7e7fc301893c27a924c73 Author: Mark Payne Date: 2018-10-05T16:06:39Z NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary ---
[GitHub] nifi pull request #3047: NIFI-5663: Ensure that when sort Node Identifiers t...
Github user markap14 closed the pull request at: https://github.com/apache/nifi/pull/3047 ---
[jira] [Commented] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639990#comment-16639990
]
ASF GitHub Bot commented on NIFI-5663:
--
Github user markap14 closed the pull request at:
https://github.com/apache/nifi/pull/3047
> FlowFile load balancing keeps re-partitioning
> -
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Koji Kawamura
>Assignee: Mark Payne
>Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node (nifi0). Using existing
> multi-node clusters don't reproduce the issue.
> # Create GenerateFlowFile -> LogAttribute
> # Then set 'Partition by attribute' LB strategy at the connection
> # Add 2nd node, nifi1
> # Generate some FlowFiles. Then load-balance activity never finishes.
> With a 2-node cluster, for some reason, each node ended up having different
> queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug
> logs, I found each node has followings:
> * nifi0
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi1.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> * nifi1
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi0.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
> received FlowFiles between each other in case the calculated attribute value
> hash points to queuePartitions[0]. Following log is written endlessly:
> {code:java}
> 2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
> o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
> from Peer: ...offset=7452,
> length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
> FlowFiles to ensure proper balancing across the cluster.
> {code}
> SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
> cluster topology change using ClusterTopologyEventListener.
> SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log
> shows the array was empty when the 2nd node (nifi1) is added:
> {code:java}
> ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
> Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
> Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from
> [] to [nifi1.example.com:8080]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5654) NiFi Documentation is shown as insecure in Firefox 62
[ https://issues.apache.org/jira/browse/NIFI-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser resolved NIFI-5654. - Resolution: Resolved Will be resolved when version 1.8.0 is released. > NiFi Documentation is shown as insecure in Firefox 62 > - > > Key: NIFI-5654 > URL: https://issues.apache.org/jira/browse/NIFI-5654 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Julian Feinauer >Priority: Major > Attachments: image-2018-10-01-20-41-16-531.png > > > While Browsing the documentation I observed that my Browser (Firefox 62) > lists the Page as insecure when going to https://nifi.apache.org/docs.html > and Clicking on "User Guide", see Screenshot attached. > !image-2018-10-01-20-41-16-531.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5654) NiFi Documentation is shown as insecure in Firefox 62
[ https://issues.apache.org/jira/browse/NIFI-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639982#comment-16639982 ] Michael Moser commented on NIFI-5654: - Thanks for reporting this! Looks to me like this is caused by a link to a missing image. It's fixed by NIFI-5330. The fix will be deployed when the 1.8.0 release is published. > NiFi Documentation is shown as insecure in Firefox 62 > - > > Key: NIFI-5654 > URL: https://issues.apache.org/jira/browse/NIFI-5654 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Julian Feinauer >Priority: Major > Attachments: image-2018-10-01-20-41-16-531.png > > > While Browsing the documentation I observed that my Browser (Firefox 62) > lists the Page as insecure when going to https://nifi.apache.org/docs.html > and Clicking on "User Guide", see Screenshot attached. > !image-2018-10-01-20-41-16-531.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5537) Create Neo4J cypher execution processor
[
https://issues.apache.org/jira/browse/NIFI-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639953#comment-16639953
]
ASF GitHub Bot commented on NIFI-5537:
--
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223045802
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user. A dummy non-blank
password is required even if it disabled on the server.")
+.required(true)
+.sensitive(true)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new
[jira] [Commented] (NIFI-5531) TCPListen issue
[ https://issues.apache.org/jira/browse/NIFI-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639956#comment-16639956 ] Bryan Bende commented on NIFI-5531: --- ListenTCP currently is meant to read new-line delimited text, so its reading the stream of incoming data and whenever a new line is encountered it takes whatever it has read up to that point and treats it as a logical message. Each logical message gets written to a flow file depending on how you configured the Batch Size, meaning if Batch Size is 1 then each line gets put in it's own flow file, if Batch Size is 100 then 100 lines would get put into a single flow file. The Outgoing Message Delimiter is what will be written between lines when Batch Size is > 1. > TCPListen issue > --- > > Key: NIFI-5531 > URL: https://issues.apache.org/jira/browse/NIFI-5531 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.7.4 > Environment: RHEL6 server to RHEL6 server over internal network >Reporter: Charles Lassiter >Priority: Major > > When utilizing the postTCP and ListenTCP, I'm receiving chunked files when > sending one text file. So as an example, If I send one file via TCP, the > ListenTCP receives the file and chunks it into 8+ files. Delimiter is set the > same on both ends so I don't understand why the ListenTCP is chunking the > files. Any suggestions would help. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5537) Create Neo4J cypher execution processor
[
https://issues.apache.org/jira/browse/NIFI-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639949#comment-16639949
]
ASF GitHub Bot commented on NIFI-5537:
--
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223045178
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue
[jira] [Commented] (NIFI-5537) Create Neo4J cypher execution processor
[
https://issues.apache.org/jira/browse/NIFI-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639955#comment-16639955
]
ASF GitHub Bot commented on NIFI-5537:
--
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223046927
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
--- End diff --
My thought was that the query results will be the flow file content, while
preserving the query as an attribute. This way down stream processors can have
access to both the results and the query for additional processing.
> Create Neo4J cypher execution processor
> ---
>
> Key: NIFI-5537
> URL: https://issues.apache.org/jira/browse/NIFI-5537
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
> Labels: graph, neo4j, node, relationship
>
> Create Nifi Neo4J cypher queries processor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223046927
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
--- End diff --
My thought was that the query results will be the flow file content, while
preserving the query as an attribute. This way down stream processors can have
access to both the results and the query for additional processing.
---
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223045178
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.name(), "Least
Connected", "Least Connected Strategy");
+
+protected static final PropertyDescriptor LOAD_BALANCING_STRATEGY =
new PropertyDescriptor.Builder()
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223045802
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user. A dummy non-blank
password is required even if it disabled on the server.")
+.required(true)
+.sensitive(true)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.name(), "Least
Connected", "Least Connected Strategy");
+
+protected static final PropertyDescriptor LOAD_BALANCING_STRATEGY =
new PropertyDescriptor.Builder()
+
[jira] [Commented] (NIFI-3993) Upgrade embedded ZooKeeper version
[ https://issues.apache.org/jira/browse/NIFI-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639948#comment-16639948 ] Michael Moser commented on NIFI-3993: - If it's possible to upgrade the embedded ZK to 3.4.10, yet keep NiFi compatible with external ZK 3.4.6, then perhaps that's a valid compromise. > Upgrade embedded ZooKeeper version > -- > > Key: NIFI-3993 > URL: https://issues.apache.org/jira/browse/NIFI-3993 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.2.0 >Reporter: Mark Bean >Priority: Major > > In a Cluster configuration, Nodes are periodically disconnected from the > Cluster, and then reconnected. These events correspond to the following error: > ERROR [CommitProcessor:1] o.apache.zookeeper.server.NIOServerCnxn Unexpected > Exception: > java.nio.channels.CancelledKeyException: null > at sun.nio.ch.SelectionKeyImpl.ensureValid(SectionKeyImpl.java:73) > at sun.nio.ch.SelectionKeyImpl.interestOps(SelctionKeyImpl.java:77) > at > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) > at > org.apache.zookeeper.server.NIOServerCnXn.sendResopnse(NIOServerCnxn.java:1081) > at > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) > at > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) > This error was reported in ZooKeeper JIRA [1], and reported as fixed in > version 3.4.10, the current stable build. As additional confirmation, when > using a stand-alone ZK, 3.4.10, rather than the embedded ZK, the above error > was no longer observed. > Update NiFi to use ZK 3.4.10 > [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2044 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5537) Create Neo4J cypher execution processor
[
https://issues.apache.org/jira/browse/NIFI-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639944#comment-16639944
]
ASF GitHub Bot commented on NIFI-5537:
--
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223044277
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
--- End diff --
@mattyb149 - Neo4j defaults to and prefers enforcing authentication. But
let me know if you think we should keep it optional.
> Create Neo4J cypher execution processor
> ---
>
> Key: NIFI-5537
> URL: https://issues.apache.org/jira/browse/NIFI-5537
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
> Labels: graph, neo4j, node, relationship
>
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user mans2singh commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r223044277
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
--- End diff --
@mattyb149 - Neo4j defaults to and prefers enforcing authentication. But
let me know if you think we should keep it optional.
---
[GitHub] nifi pull request #3047: NIFI-5663: Ensure that when sort Node Identifiers t...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3047 NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5663 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3047.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3047 commit 619f1ffe8fbbca61bc5545f13920190a77006e08 Author: Mark Payne Date: 2018-06-14T15:57:21Z NIFI-5516: Implement Load-Balanced Connections Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue Refactored more into AbstractFlowFileQueue Added documentation, cleaned up code some Refactored FlowFileQueue so that there is SwappablePriorityQueue Several unit tests written Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles. Bug fixes Code refactoring Added integration tests, bug fixes Refactored clients to use NIO Bug fixes. Appears to finally be working with NIO Client! NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession Bug fixes and allowed load balancing socket connections to be reused Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing Added flag to ConnectionDTO to indicate Load Balance Status Updated Diagnostics DTO for connections Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart Code cleanup Fixed checkstyle and unit tests NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is. NIFI-5516: FIxed missing License headers NIFI-5516: Some minor code cleanup NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release NIFI-5516: Take backpressure configuration into account NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier NIFI-5516: Addressed review feedback This closes #2947 commit 47bbe20f9ee3c348378adfa86965f34a319057bb Author: Mark Payne Date: 2018-10-05T15:12:49Z NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary ---
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user mans2singh commented on a diff in the pull request: https://github.com/apache/nifi/pull/2956#discussion_r223043551 --- Diff: nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/pom.xml --- @@ -0,0 +1,87 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-neo4j-bundle +1.8.0-SNAPSHOT + + +nifi-neo4j-processors +jar + + + + org.neo4j.driver + neo4j-java-driver + 1.6.2 + compile + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-utils +1.8.0-SNAPSHOT + + +commons-io +commons-io +2.6 + + +org.apache.commons +commons-lang3 +3.7 + + +com.google.code.gson +gson +2.7 + + +org.apache.nifi +nifi-mock +1.8.0-SNAPSHOT +test + + +com.google.guava +guava +18.0 --- End diff -- Will work on it. ---
[jira] [Commented] (NIFI-5537) Create Neo4J cypher execution processor
[ https://issues.apache.org/jira/browse/NIFI-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639941#comment-16639941 ] ASF GitHub Bot commented on NIFI-5537: -- Github user mans2singh commented on a diff in the pull request: https://github.com/apache/nifi/pull/2956#discussion_r223043551 --- Diff: nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/pom.xml --- @@ -0,0 +1,87 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-neo4j-bundle +1.8.0-SNAPSHOT + + +nifi-neo4j-processors +jar + + + + org.neo4j.driver + neo4j-java-driver + 1.6.2 + compile + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-utils +1.8.0-SNAPSHOT + + +commons-io +commons-io +2.6 + + +org.apache.commons +commons-lang3 +3.7 + + +com.google.code.gson +gson +2.7 + + +org.apache.nifi +nifi-mock +1.8.0-SNAPSHOT +test + + +com.google.guava +guava +18.0 --- End diff -- Will work on it. > Create Neo4J cypher execution processor > --- > > Key: NIFI-5537 > URL: https://issues.apache.org/jira/browse/NIFI-5537 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.7.1 > Environment: All >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: graph, neo4j, node, relationship > > Create Nifi Neo4J cypher queries processor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[ https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639938#comment-16639938 ] ASF GitHub Bot commented on NIFI-5663: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3047 NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5663 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3047.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3047 commit 619f1ffe8fbbca61bc5545f13920190a77006e08 Author: Mark Payne Date: 2018-06-14T15:57:21Z NIFI-5516: Implement Load-Balanced Connections Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue Refactored more into AbstractFlowFileQueue Added documentation, cleaned up code some Refactored FlowFileQueue so that there is SwappablePriorityQueue Several unit tests written Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles. Bug fixes Code refactoring Added integration tests, bug fixes Refactored clients to use NIO Bug fixes. Appears to finally be working with NIO Client! NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession Bug fixes and allowed load balancing socket connections to be reused Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing Added flag to ConnectionDTO to indicate Load Balance Status Updated Diagnostics DTO for connections Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart Code cleanup Fixed checkstyle and unit tests NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is. NIFI-5516: FIxed missing License headers NIFI-5516: Some minor code cleanup NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release NIFI-5516: Take backpressure configuration into account NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier NIFI-5516: Addressed review feedback This closes #2947 commit 47bbe20f9ee3c348378adfa86965f34a319057bb Author: Mark Payne Date: 2018-10-05T15:12:49Z NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when
[jira] [Updated] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-5663:
-
Status: Patch Available (was: Open)
> FlowFile load balancing keeps re-partitioning
> -
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Koji Kawamura
>Assignee: Mark Payne
>Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node (nifi0). Using existing
> multi-node clusters don't reproduce the issue.
> # Create GenerateFlowFile -> LogAttribute
> # Then set 'Partition by attribute' LB strategy at the connection
> # Add 2nd node, nifi1
> # Generate some FlowFiles. Then load-balance activity never finishes.
> With a 2-node cluster, for some reason, each node ended up having different
> queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug
> logs, I found each node has followings:
> * nifi0
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi1.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> * nifi1
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi0.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
> received FlowFiles between each other in case the calculated attribute value
> hash points to queuePartitions[0]. Following log is written endlessly:
> {code:java}
> 2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
> o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
> from Peer: ...offset=7452,
> length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
> FlowFiles to ensure proper balancing across the cluster.
> {code}
> SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
> cluster topology change using ClusterTopologyEventListener.
> SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log
> shows the array was empty when the 2nd node (nifi1) is added:
> {code:java}
> ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
> Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
> Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from
> [] to [nifi1.example.com:8080]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3993) Upgrade embedded ZooKeeper version
[ https://issues.apache.org/jira/browse/NIFI-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639903#comment-16639903 ] ASF GitHub Bot commented on NIFI-3993: -- Github user HorizonNet commented on the issue: https://github.com/apache/nifi/pull/2884 @jtstorck If most of the services/components depend on ZooKeeper 3.4.6 then let's stick with that version and close the ticket. Any objections? > Upgrade embedded ZooKeeper version > -- > > Key: NIFI-3993 > URL: https://issues.apache.org/jira/browse/NIFI-3993 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.2.0 >Reporter: Mark Bean >Priority: Major > > In a Cluster configuration, Nodes are periodically disconnected from the > Cluster, and then reconnected. These events correspond to the following error: > ERROR [CommitProcessor:1] o.apache.zookeeper.server.NIOServerCnxn Unexpected > Exception: > java.nio.channels.CancelledKeyException: null > at sun.nio.ch.SelectionKeyImpl.ensureValid(SectionKeyImpl.java:73) > at sun.nio.ch.SelectionKeyImpl.interestOps(SelctionKeyImpl.java:77) > at > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) > at > org.apache.zookeeper.server.NIOServerCnXn.sendResopnse(NIOServerCnxn.java:1081) > at > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) > at > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) > This error was reported in ZooKeeper JIRA [1], and reported as fixed in > version 3.4.10, the current stable build. As additional confirmation, when > using a stand-alone ZK, 3.4.10, rather than the embedded ZK, the above error > was no longer observed. > Update NiFi to use ZK 3.4.10 > [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2044 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2884: NIFI-3993 Updated the ZooKeeper version to 3.4.10
Github user HorizonNet commented on the issue: https://github.com/apache/nifi/pull/2884 @jtstorck If most of the services/components depend on ZooKeeper 3.4.6 then let's stick with that version and close the ticket. Any objections? ---
[jira] [Commented] (NIFIREG-205) NiFi Registry DB gets out of sync with git repository, no apprent remediation
[ https://issues.apache.org/jira/browse/NIFIREG-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639879#comment-16639879 ] Bryan Bende commented on NIFIREG-205: - [~ijokarumawak] might be interested since he worked on the git integration. > NiFi Registry DB gets out of sync with git repository, no apprent remediation > - > > Key: NIFIREG-205 > URL: https://issues.apache.org/jira/browse/NIFIREG-205 > Project: NiFi Registry > Issue Type: Bug > Environment: Centos 7.5 >Reporter: Dye357 >Priority: Major > > I've observed a couple issues with the GitFlowPersistenceAdapter: > # When adding a new process group to NIFREG If for any reason the git > repository is in a "dirty" (untracked file) state the adding of the process > group fails. However an entry is still created in the DB with a version of 0. > Once in this state you cannot delete the flow from NIFIREG and you cannot > restart version control from nifi with the same name. I assume the only way > to fix this is to manually go into the DB and delete the record. > # When using Remote To Push, if the push fails the same behavior in #1 is > exhibited. It's not reasonable to expect that a push will always succeed. The > remote git repository could be offline for maintenance etc... > Steps to reproduce: > # Start nifi registry with an empty db and clean git repo. > # add an untracked file to the git repo but do-not commit it. > # Start a processgroup under version control. > # Expect Failure in Nifi UI > # Expect Exception in Log saying untracked files in git repo. > # Delete flow from nifi-registry using Actions -> Delete. > # Expect Failure case, recieve error deleting flow message. > # Refresh nifi-registry UI - flow is still present, version is 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-205) NiFi Registry DB gets out of sync with git repository, no apprent remediation
[ https://issues.apache.org/jira/browse/NIFIREG-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639878#comment-16639878 ] Bryan Bende commented on NIFIREG-205: - I do agree that this should be improved, but for #1, is itreally a valid scenario? No one should be manually modifying the git repo, so is there any way to get an untracked file from using the application? > NiFi Registry DB gets out of sync with git repository, no apprent remediation > - > > Key: NIFIREG-205 > URL: https://issues.apache.org/jira/browse/NIFIREG-205 > Project: NiFi Registry > Issue Type: Bug > Environment: Centos 7.5 >Reporter: Dye357 >Priority: Major > > I've observed a couple issues with the GitFlowPersistenceAdapter: > # When adding a new process group to NIFREG If for any reason the git > repository is in a "dirty" (untracked file) state the adding of the process > group fails. However an entry is still created in the DB with a version of 0. > Once in this state you cannot delete the flow from NIFIREG and you cannot > restart version control from nifi with the same name. I assume the only way > to fix this is to manually go into the DB and delete the record. > # When using Remote To Push, if the push fails the same behavior in #1 is > exhibited. It's not reasonable to expect that a push will always succeed. The > remote git repository could be offline for maintenance etc... > Steps to reproduce: > # Start nifi registry with an empty db and clean git repo. > # add an untracked file to the git repo but do-not commit it. > # Start a processgroup under version control. > # Expect Failure in Nifi UI > # Expect Exception in Log saying untracked files in git repo. > # Delete flow from nifi-registry using Actions -> Delete. > # Expect Failure case, recieve error deleting flow message. > # Refresh nifi-registry UI - flow is still present, version is 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne reassigned NIFI-5663:
Assignee: Mark Payne
> FlowFile load balancing keeps re-partitioning
> -
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Koji Kawamura
>Assignee: Mark Payne
>Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node (nifi0). Using existing
> multi-node clusters don't reproduce the issue.
> # Create GenerateFlowFile -> LogAttribute
> # Then set 'Partition by attribute' LB strategy at the connection
> # Add 2nd node, nifi1
> # Generate some FlowFiles. Then load-balance activity never finishes.
> With a 2-node cluster, for some reason, each node ended up having different
> queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug
> logs, I found each node has followings:
> * nifi0
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi1.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> * nifi1
> ** queuePartitions[0} =
> RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
> nodeId=nifi0.example.com:8080]
> ** queuePartitions[1} =
> SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
> Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
> received FlowFiles between each other in case the calculated attribute value
> hash points to queuePartitions[0]. Following log is written endlessly:
> {code:java}
> 2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
> o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
> from Peer: ...offset=7452,
> length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
> FlowFiles to ensure proper balancing across the cluster.
> {code}
> SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
> cluster topology change using ClusterTopologyEventListener.
> SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log
> shows the array was empty when the 2nd node (nifi1) is added:
> {code:java}
> ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
> Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
> Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from
> [] to [nifi1.example.com:8080]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (MINIFICPP-615) Provide Windows OpenSSL Support
[ https://issues.apache.org/jira/browse/MINIFICPP-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-615: - Description: Ensure OpenSSL support works throughout our project in windows. > Provide Windows OpenSSL Support > --- > > Key: MINIFICPP-615 > URL: https://issues.apache.org/jira/browse/MINIFICPP-615 > Project: NiFi MiNiFi C++ > Issue Type: Bug > Environment: Microsoft Windows >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: windows > Fix For: 0.6.0 > > > Ensure OpenSSL support works throughout our project in windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-615) Provide Windows OpenSSL Support
[ https://issues.apache.org/jira/browse/MINIFICPP-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-615: - Labels: windows (was: ) > Provide Windows OpenSSL Support > --- > > Key: MINIFICPP-615 > URL: https://issues.apache.org/jira/browse/MINIFICPP-615 > Project: NiFi MiNiFi C++ > Issue Type: Bug > Environment: Microsoft Windows >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: windows > Fix For: 0.6.0 > > > Ensure OpenSSL support works throughout our project in windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5664) LookupRecord cannot handle JSON array properties from MongoDBLookupService
Carl Gieringer created NIFI-5664:
Summary: LookupRecord cannot handle JSON array properties from
MongoDBLookupService
Key: NIFI-5664
URL: https://issues.apache.org/jira/browse/NIFI-5664
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.7.1, 1.8.0
Reporter: Carl Gieringer
We are using a LookupRecord processor to add an array property to a Record.
The LookupRecord's `Lookup Service` is a MongoDBLookupService. The idea is for
the LookupRecord to query Mongo documents by the Record's `_id`, to extract the
array property at `f` on that document, and to insert that array into the
Record at the RecordPath `f`. This flow results in the conversion error copied
below, which completely prevents the FlowFile from proceeding (routes to
Failure/Unmatched.) It appears that the `MongoDBLookupService` (or perhaps its
`MongoDBControllerService`) stores Mongo document arrays as `ArrayList`s, which
`DataTypeUtils#toArray` does not support. A simple fix appears to be to add a
few lines to `DataTypeUtils#toArray` to support `ArrayList`s. I will provide a
PR for this issue.
This issue appears in 1.7.1, and is extant in 1.8.0-SNAPSHOT (2018-10-02
b4c8e0179). I am not aware of any workaround to enrich a Record with an array
from Mongo.
LookupRecord Configuration
||Property||Value||
|Result RecordPath|/f|
|Record Result Contents|Insert Entire Record|
|_id|/_id|
MongoDBLookupService Configuration
||Property||Value||
|Lookup Value Field|f|
Conversion error (with irrelevant Mongo document and schema properties elided):
{code:java}
2018-10-04 15:00:31,348 ERROR [Timer-Driven Process Thread-1]
o.a.n.processors.standard.LookupRecord
LookupRecord[id=339bf72c-91c3-36f0-ac14-bc8b734e2582] Failed to write
MapRecord[{... f=[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
Document{{fid=5762adf32fc74d0418104201, l=2.12}},
Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
Document{{fid=587542dd638e870838a79d08, l=1.0}},
Document{{fid=5762addbffa7cf0980267acd, l=3.32}}],
i=[Ljava.lang.Object;@35dee8fc, ...] with schema {
"namespace": "nifi",
"name": "MongoPosting",
"type": "record",
"fields": [
...
{ "name": "f", "type": [
"null",
{ "type": "array", "items": {
"type": "record", "name": "PostingFeatureDetails", "fields": [
{ "name": "fid", "type": "string" },
{ "name": "l" , "type": ["null", "double"] }
]}
}
]},
...
]
} as a JSON Object due to
org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
Document{{fid=5762adf32fc74d0418104201, l=2.12}},
Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
Document{{fid=587542dd638e870838a79d08, l=1.0}},
Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
java.util.ArrayList to Object Array for field f:
org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
Document{{fid=5762adf32fc74d0418104201, l=2.12}},
Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
Document{{fid=587542dd638e870838a79d08, l=1.0}},
Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
java.util.ArrayList to Object Array for field f
org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}},
Document{{fid=5762adf32fc74d0418104201, l=2.12}},
Document{{fid=5762addd18b8b607802fec6f, l=3.32}},
Document{{fid=587542dd638e870838a79d08, l=1.0}},
Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class
java.util.ArrayList to Object Array for field f
at
org.apache.nifi.serialization.record.util.DataTypeUtils.toArray(DataTypeUtils.java:347)
at
org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:153)
at
org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:115)
at org.apache.nifi.json.WriteJsonResult.writeValue(WriteJsonResult.java:284)
at
org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:187)
at
org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:136)
at
org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
at
org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:148)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
at
org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
at
[jira] [Commented] (NIFI-5661) Add UI support for load balancing connections
[ https://issues.apache.org/jira/browse/NIFI-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639780#comment-16639780 ] ASF GitHub Bot commented on NIFI-5661: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3046 Will review... > Add UI support for load balancing connections > - > > Key: NIFI-5661 > URL: https://issues.apache.org/jira/browse/NIFI-5661 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > NIFI-5516 adds FlowFile load-balancing feature at connections. We need UI > controls to: > - Configure if a connection is load balanced and what strategy to use. > - Indicator that load balancing is in progress -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3046: NIFI-5661: Adding Load Balance config UI
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3046 Will review... ---
[jira] [Created] (NIFIREG-205) NiFi Registry DB gets out of sync with git repository, no apprent remediation
Dye357 created NIFIREG-205: -- Summary: NiFi Registry DB gets out of sync with git repository, no apprent remediation Key: NIFIREG-205 URL: https://issues.apache.org/jira/browse/NIFIREG-205 Project: NiFi Registry Issue Type: Bug Environment: Centos 7.5 Reporter: Dye357 I've observed a couple issues with the GitFlowPersistenceAdapter: # When adding a new process group to NIFREG If for any reason the git repository is in a "dirty" (untracked file) state the adding of the process group fails. However an entry is still created in the DB with a version of 0. Once in this state you cannot delete the flow from NIFIREG and you cannot restart version control from nifi with the same name. I assume the only way to fix this is to manually go into the DB and delete the record. # When using Remote To Push, if the push fails the same behavior in #1 is exhibited. It's not reasonable to expect that a push will always succeed. The remote git repository could be offline for maintenance etc... Steps to reproduce: # Start nifi registry with an empty db and clean git repo. # add an untracked file to the git repo but do-not commit it. # Start a processgroup under version control. # Expect Failure in Nifi UI # Expect Exception in Log saying untracked files in git repo. # Delete flow from nifi-registry using Actions -> Delete. # Expect Failure case, recieve error deleting flow message. # Refresh nifi-registry UI - flow is still present, version is 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5663:
Description:
Scenario
# Start a brand-new cluster with only 1 node. Using existing multi-node
clusters don't reproduce the issue. nifi0
# Create GenerateFlowFile -> LogAttribute
# Then set 'Partition by attribute' LB strategy at the connection
# Add 2nd node, nifi1
# Generate some FlowFiles. Then load-balance activity never finishes.
With a 2-node cluster, for some reason, each node ended up having different
queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs,
I found each node has followings:
* nifi0
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi1.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
* nifi1
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi0.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
received FlowFiles between each other in case the calculated attribute value
hash points to queuePartitions[0]. Following log is written endlessly:
{code:java}
2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
from Peer: ...offset=7452,
length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
FlowFiles to ensure proper balancing across the cluster.
{code}
SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
cluster topology change using ClusterTopologyEventListener.
SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows
the array was empty when the 2nd node (nifi1) is added:
{code:java}
ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from []
to [nifi1.example.com:8080]
{code}
was:
Scenario
# Start a brand-new cluster with only 1 node. nifi0
# Create GenerateFlowFile -> LogAttribute
# Then set 'Partition by attribute' LB strategy at the connection
# Add 2nd node, nifi1
# Generate some FlowFiles. Then load-balance activity never finishes.
With a 2-node cluster, for some reason, each node ended up having different
queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs,
I found each node has followings:
* nifi0
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi1.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
* nifi1
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi0.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
received FlowFiles between each other in case the calculated attribute value
hash points to queuePartitions[0]. Following log is written endlessly:
{code:java}
2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
from Peer: ...offset=7452,
length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
FlowFiles to ensure proper balancing across the cluster.
{code}
SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
cluster topology change using ClusterTopologyEventListener.
SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows
the array was empty when the 2nd node (nifi1) is added:
{code:java}
ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from []
to [nifi1.example.com:8080]
{code}
> FlowFile load balancing keeps re-partitioning
> -
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Koji Kawamura
>Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node. Using existing multi-node
> clusters don't reproduce the issue. nifi0
> # Create GenerateFlowFile -> LogAttribute
> # Then set 'Partition by attribute' LB strategy
[jira] [Updated] (NIFI-5663) FlowFile load balancing keeps re-partitioning
[
https://issues.apache.org/jira/browse/NIFI-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5663:
Description:
Scenario
# Start a brand-new cluster with only 1 node (nifi0). Using existing
multi-node clusters don't reproduce the issue.
# Create GenerateFlowFile -> LogAttribute
# Then set 'Partition by attribute' LB strategy at the connection
# Add 2nd node, nifi1
# Generate some FlowFiles. Then load-balance activity never finishes.
With a 2-node cluster, for some reason, each node ended up having different
queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs,
I found each node has followings:
* nifi0
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi1.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
* nifi1
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi0.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
received FlowFiles between each other in case the calculated attribute value
hash points to queuePartitions[0]. Following log is written endlessly:
{code:java}
2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
from Peer: ...offset=7452,
length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
FlowFiles to ensure proper balancing across the cluster.
{code}
SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
cluster topology change using ClusterTopologyEventListener.
SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows
the array was empty when the 2nd node (nifi1) is added:
{code:java}
ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from []
to [nifi1.example.com:8080]
{code}
was:
Scenario
# Start a brand-new cluster with only 1 node. Using existing multi-node
clusters don't reproduce the issue. nifi0
# Create GenerateFlowFile -> LogAttribute
# Then set 'Partition by attribute' LB strategy at the connection
# Add 2nd node, nifi1
# Generate some FlowFiles. Then load-balance activity never finishes.
With a 2-node cluster, for some reason, each node ended up having different
queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs,
I found each node has followings:
* nifi0
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi1.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
* nifi1
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi0.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
received FlowFiles between each other in case the calculated attribute value
hash points to queuePartitions[0]. Following log is written endlessly:
{code:java}
2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
from Peer: ...offset=7452,
length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
FlowFiles to ensure proper balancing across the cluster.
{code}
SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
cluster topology change using ClusterTopologyEventListener.
SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows
the array was empty when the 2nd node (nifi1) is added:
{code:java}
ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from []
to [nifi1.example.com:8080]
{code}
> FlowFile load balancing keeps re-partitioning
> -
>
> Key: NIFI-5663
> URL: https://issues.apache.org/jira/browse/NIFI-5663
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Koji Kawamura
>Priority: Critical
>
> Scenario
> # Start a brand-new cluster with only 1 node (nifi0). Using existing
> multi-node clusters don't reproduce the issue.
> # Create GenerateFlowFile
[jira] [Created] (NIFI-5663) FlowFile load balancing keeps re-partitioning
Koji Kawamura created NIFI-5663:
---
Summary: FlowFile load balancing keeps re-partitioning
Key: NIFI-5663
URL: https://issues.apache.org/jira/browse/NIFI-5663
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.8.0
Reporter: Koji Kawamura
Scenario
# Start a brand-new cluster with only 1 node. nifi0
# Create GenerateFlowFile -> LogAttribute
# Then set 'Partition by attribute' LB strategy at the connection
# Add 2nd node, nifi1
# Generate some FlowFiles. Then load-balance activity never finishes.
With a 2-node cluster, for some reason, each node ended up having different
queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs,
I found each node has followings:
* nifi0
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi1.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
* nifi1
** queuePartitions[0} =
RemoteQueuePartition[queueId=14ac9634-0166-1000--d9ae7f4b,
nodeId=nifi0.example.com:8080]
** queuePartitions[1} =
SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000--d9ae7f4b]
Because of this, 'Partition by attribute' LB strategy keeps re-partitioning
received FlowFiles between each other in case the calculated attribute value
hash points to queuePartitions[0]. Following log is written endlessly:
{code:java}
2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles
from Peer: ...offset=7452,
length=180],offset=162,name=10653317458635,size=18]]. Will re-partition
FlowFiles to ensure proper balancing across the cluster.
{code}
SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to
cluster topology change using ClusterTopologyEventListener.
SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows
the array was empty when the 2nd node (nifi1) is added:
{code:java}
ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process
Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node
Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from []
to [nifi1.example.com:8080]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5662) AvroTypeUtil Decimal support using Fixed Error
Gideon Korir created NIFI-5662:
--
Summary: AvroTypeUtil Decimal support using Fixed Error
Key: NIFI-5662
URL: https://issues.apache.org/jira/browse/NIFI-5662
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.7.1
Environment: RHEL 7.5
JDK 1.8.182
Reporter: Gideon Korir
When the decimal is specified as fixed in the Avro schema, AvroTypeUtils
converts the decimal into a ByteBuffer instead of a GenericFixed.
The code:
{code:java}
return new Conversions.DecimalConversion().toBytes(decimal, fieldSchema,
logicalType)
{code}
Should be:
{code:java}
return fieldSchema.getType() == Type.BYTES
? new Conversions.DecimalConversion().toBytes(decimal, fieldSchema, logicalType)
: new Conversions.DecimalConversion().toFixed(decimal, fieldSchema,
logicalType);
{code}
The former causes the AvroRecordSetWriter to fail with the error:
_org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to
org.apache.avro.generic.GenericFixed_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
