[jira] [Commented] (MINIFICPP-652) Add monitored threads to stacks protocol
[ https://issues.apache.org/jira/browse/MINIFICPP-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665778#comment-16665778 ] ASF GitHub Bot commented on MINIFICPP-652: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/428 MINIFICPP-652: Add traceability to repositories and flow controller Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-652-b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/428.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #428 commit 5fe24701cbe690e9bc2df95ea638271d22cf8812 Author: Marc Parisi Date: 2018-10-22T17:21:46Z MINIFICPP-652: Add traceability to repositories and flow controller > Add monitored threads to stacks protocol > > > Key: MINIFICPP-652 > URL: https://issues.apache.org/jira/browse/MINIFICPP-652 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #428: MINIFICPP-652: Add traceability to reposi...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/428 MINIFICPP-652: Add traceability to repositories and flow controller Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-652-b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/428.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #428 commit 5fe24701cbe690e9bc2df95ea638271d22cf8812 Author: Marc Parisi Date: 2018-10-22T17:21:46Z MINIFICPP-652: Add traceability to repositories and flow controller ---
[jira] [Commented] (NIFI-5318) Implement NiFi test harness
[
https://issues.apache.org/jira/browse/NIFI-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665758#comment-16665758
]
ASF GitHub Bot commented on NIFI-5318:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2872#discussion_r228689270
--- Diff:
nifi-testharness/src/test/java/org/apache/nifi/testharness/samples/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.nifi.testharness.samples;
+
+import java.io.File;
+
+public final class Constants {
+
+static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
+
+// NOTE: you will have to have the NiFi distribution ZIP placed into
this directory.
+// Its version must be the same as the one referenced in the flow.xml,
otherwise it will not work!
+static final File NIFI_ZIP_DIR = new
File(System.getProperty("user.home"), "Downloads");
--- End diff --
That folder evaluated to `$HOME/Downloads` for my user account when I ran
it. Are you expecting the user to sudo or something?
> Implement NiFi test harness
> ---
>
> Key: NIFI-5318
> URL: https://issues.apache.org/jira/browse/NIFI-5318
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Peter Horvath
>Priority: Major
>
> Currently, it is not really possible to automatically test the behaviour of a
> specific NiFi flow and make unit test type asserts if it works as expected.
> For example, if the expected behaviour of a NiFi flow is that a file placed
> to a specific directory will trigger some operation after which some output
> file will appear at another directory, once currently can only do one thing:
> test the NiFi flow manually.
> Manual testing is especially hard to manage if a NiFi flow is being actively
> developed: any change to a complex, existing NiFi flow might require a lot of
> manual testing just to ensure there are no regressions introduced.
> Some kind of Java API that allows managing a NiFi instance and manipulating
> flow deployments like for example, [Codehaus
> Cargo|]https://codehaus-cargo.github.io/] would be of great help.
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2872: NIFI-5318 Implement NiFi test harness: initial comm...
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2872#discussion_r228689270
--- Diff:
nifi-testharness/src/test/java/org/apache/nifi/testharness/samples/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.nifi.testharness.samples;
+
+import java.io.File;
+
+public final class Constants {
+
+static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
+
+// NOTE: you will have to have the NiFi distribution ZIP placed into
this directory.
+// Its version must be the same as the one referenced in the flow.xml,
otherwise it will not work!
+static final File NIFI_ZIP_DIR = new
File(System.getProperty("user.home"), "Downloads");
--- End diff --
That folder evaluated to `$HOME/Downloads` for my user account when I ran
it. Are you expecting the user to sudo or something?
---
[jira] [Commented] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[ https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665625#comment-16665625 ] ASF GitHub Bot commented on NIFI-5724: -- GitHub user viswaug opened a pull request: https://github.com/apache/nifi/pull/3113 making the database connection autocommit value a configurable property Currently, the PutSQL processor sets the database connection auto commit to false before executing the SQL statement(s) and this is hardcoded in. This commit makes the auto commit value being set to be configurable. https://issues.apache.org/jira/projects/NIFI/issues/NIFI-5724?filter=allissues You can merge this pull request into a Git repository by running: $ git pull https://github.com/Snagajob/nifi configurable_autocommit_putsql Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3113 commit f63314c3db909d99acce1a071faa889a9cceca36 Author: Vish Uma Date: 2018-10-26T19:32:46Z making the database session autocommit value a configurable property > Make the autocommit value in the PutSQL processor configurable > -- > > Key: NIFI-5724 > URL: https://issues.apache.org/jira/browse/NIFI-5724 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: vish uma >Priority: Minor > > The PutSQL processor currently always sets the autocommit value on the > database session to false before the SQL statement is run and resets it back > to the original value after. > i am not sure if the autocommit value is hardcoded to false for a reason, if > it is, please let me know. > This is causing an issue with the snowflake DB where abruptly disconnected > sessions do not release the locks they have taken. > i would like to make this autocommit value configurable. I can submit a patch > for this if there is no objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3113: making the database connection autocommit value a c...
GitHub user viswaug opened a pull request: https://github.com/apache/nifi/pull/3113 making the database connection autocommit value a configurable property Currently, the PutSQL processor sets the database connection auto commit to false before executing the SQL statement(s) and this is hardcoded in. This commit makes the auto commit value being set to be configurable. https://issues.apache.org/jira/projects/NIFI/issues/NIFI-5724?filter=allissues You can merge this pull request into a Git repository by running: $ git pull https://github.com/Snagajob/nifi configurable_autocommit_putsql Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3113 commit f63314c3db909d99acce1a071faa889a9cceca36 Author: Vish Uma Date: 2018-10-26T19:32:46Z making the database session autocommit value a configurable property ---
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop processing data if it evaluates invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665617#comment-16665617 ] Gardella Juan Pablo commented on NIFI-5761: --- Patch available at [https://github.com/apache/nifi/pull/3112] > ReplaceText processor can stop processing data if it evaluates invalid > expressions > -- > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.7.1 > Environment: ALL >Reporter: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop processing data if it evaluates invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665616#comment-16665616 ] ASF GitHub Bot commented on NIFI-5761: -- GitHub user gardellajuanpablo opened a pull request: https://github.com/apache/nifi/pull/3112 NIFI-5761 ReplaceText processor can stop processing data if it evalua… …tes invalid expressions Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gardellajuanpablo/nifi NIFI-5761 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3112 commit f26389e1f7f429f8fdc7f84938ff5b39658e1af1 Author: Gardella Juan Pablo Date: 2018-10-26T19:41:26Z NIFI-5761 ReplaceText processor can stop processing data if it evaluates invalid expressions > ReplaceText processor can stop processing data if it evaluates invalid > expressions > -- > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.7.1 > Environment: ALL >Reporter: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Storck resolved NIFI-5720. --- Resolution: Done > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3112: NIFI-5761 ReplaceText processor can stop processing...
GitHub user gardellajuanpablo opened a pull request: https://github.com/apache/nifi/pull/3112 NIFI-5761 ReplaceText processor can stop processing data if it evalua⦠â¦tes invalid expressions Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gardellajuanpablo/nifi NIFI-5761 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3112 commit f26389e1f7f429f8fdc7f84938ff5b39658e1af1 Author: Gardella Juan Pablo Date: 2018-10-26T19:41:26Z NIFI-5761 ReplaceText processor can stop processing data if it evaluates invalid expressions ---
[jira] [Updated] (NIFI-5760) TeradataAdapter for GenerateTableFetch and QueryDatabaseTable
[ https://issues.apache.org/jira/browse/NIFI-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Storck updated NIFI-5760: -- Fix Version/s: (was: 1.8.0) > TeradataAdapter for GenerateTableFetch and QueryDatabaseTable > - > > Key: NIFI-5760 > URL: https://issues.apache.org/jira/browse/NIFI-5760 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration >Reporter: Siddharth >Priority: Major > Labels: features > > Hi, We came across a requirement to fetch huge amount of data from Teradata > but since there are no adapter for teradata available we were forced to use > MSSQL, It generated a query but it required some modification to make it > teradata compatible. So it will be great if we can have a Teradata adapter > for GenerateTableFetch, QUeryDatabase and other JDBC component -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5760) TeradataAdapter for GenerateTableFetch and QueryDatabaseTable
[ https://issues.apache.org/jira/browse/NIFI-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665612#comment-16665612 ] Jeff Storck commented on NIFI-5760: --- Removing fix version of 1.8.0 from this JIRA. The JIRA was created after the vote for NiFi 1.8.0 RC3 was concluded. > TeradataAdapter for GenerateTableFetch and QueryDatabaseTable > - > > Key: NIFI-5760 > URL: https://issues.apache.org/jira/browse/NIFI-5760 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration >Reporter: Siddharth >Priority: Major > Labels: features > Fix For: 1.8.0 > > > Hi, We came across a requirement to fetch huge amount of data from Teradata > but since there are no adapter for teradata available we were forced to use > MSSQL, It generated a query but it required some modification to make it > teradata compatible. So it will be great if we can have a Teradata adapter > for GenerateTableFetch, QUeryDatabase and other JDBC component -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665611#comment-16665611 ] ASF subversion and git services commented on NIFI-5720: --- Commit 1844922 from jsto...@apache.org in branch 'site/trunk' [ https://svn.apache.org/r1844922 ] NIFI-5720 Added NiFi 1.8.0 documentation > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665596#comment-16665596 ] ASF subversion and git services commented on NIFI-5720: --- Commit 1844921 from jsto...@apache.org in branch 'site/trunk' [ https://svn.apache.org/r1844921 ] NIFI-5720 removing 1.7.1 docs in preparation for adding 1.8.0 docs > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5761) ReplaceText processor can stop processing data if it evaluates invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gardella Juan Pablo updated NIFI-5761: -- Summary: ReplaceText processor can stop processing data if it evaluates invalid expressions (was: ReplaceText processor can stop pulling data if the data contains invalid expressions) > ReplaceText processor can stop processing data if it evaluates invalid > expressions > -- > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.7.1 > Environment: ALL >Reporter: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5761) ReplaceText processor can stop pulling data if the data contains invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gardella Juan Pablo updated NIFI-5761: -- Affects Version/s: 1.7.1 > ReplaceText processor can stop pulling data if the data contains invalid > expressions > > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.7.1 > Environment: ALL >Reporter: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5102) MarkLogic DB Processors
[ https://issues.apache.org/jira/browse/NIFI-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665505#comment-16665505 ] Yash Tambawala commented on NIFI-5102: -- Hello Joe. Please let me know what you think about the guide and if it fits your need.. > MarkLogic DB Processors > --- > > Key: NIFI-5102 > URL: https://issues.apache.org/jira/browse/NIFI-5102 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Anthony Roach >Priority: Major > Attachments: > 0002-NIFI-5102-making-some-updates-to-help-but-security-n.patch > > > As a data architect, I need to ingest data from my NiFi FlowFile into > MarkLogic database documents. I have created the following two processors: > * PutMarkLogic: Ingest Flowfile into MarkLogic database documents > * QueryMarkLogic: Retrieve result set from MarkLogic into FlowFile > I will create a pull request. > [www.marklogic.com|http://www.marklogic.com/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665485#comment-16665485 ] ASF subversion and git services commented on NIFI-5720: --- Commit f6d8eada6a1f131345495f00f09729adcaa58cce in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f6d8ead ] NIFI-5720 Updated dockerhub module for next release > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665475#comment-16665475 ] ASF subversion and git services commented on NIFI-5720: --- Commit 4f14e517db122b56da22df050a31ac08216a8305 in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4f14e51 ] Merge branch 'NIFI-5720-RC3' as part of the NiFi 1.8.0 release process > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665474#comment-16665474 ] ASF subversion and git services commented on NIFI-5720: --- Commit c0182294ed914b9bc5fbeb4ff2b084b246e454ca in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c018229 ] NIFI-5720-RC3 prepare for next development iteration > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5720) Release Apache NiFi 1.8.0
[ https://issues.apache.org/jira/browse/NIFI-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665473#comment-16665473 ] ASF subversion and git services commented on NIFI-5720: --- Commit 98aabf2c50f857efc72fd6f2bfdd9965b97fa195 in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=98aabf2 ] NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 > Release Apache NiFi 1.8.0 > - > > Key: NIFI-5720 > URL: https://issues.apache.org/jira/browse/NIFI-5720 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Assignee: Jeff Storck >Priority: Blocker > Labels: release > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5318) Implement NiFi test harness
[
https://issues.apache.org/jira/browse/NIFI-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665467#comment-16665467
]
ASF GitHub Bot commented on NIFI-5318:
--
Github user peter-gergely-horvath commented on the issue:
https://github.com/apache/nifi/pull/2872
@MikeThomsen This is kind of tricky: NiFi flows refer the precise version
of NiFi within the flow XML file, so you have something like this for the
processor definitions:
```
org.apache.nifi.processors.standard.GetHTTP
org.apache.nifi
nifi-standard-nar
1.7.1
```
When you export a flow file to test, I would expect the you to use
precisely the same NiFi version as the one used to export the flow file.
I think there is a little bit of confusion here: please note that these
tests do not test the test-harness. The test-harness itself is not tested (kind
of a chicken-and-egg issue): these test cases are merely _samples_, which are
referenced in the documentation: they demonstrate how an _end-user_ could
create test cases for his/her own flows.
Please download the ZIP of NiFi 1.7.1 to the `Downloads` directory within
your user home (referenced by
`org.apache.nifi.testharness.samples.Constants#NIFI_ZIP_DIR` as
`new File(System.getProperty("user.home"), "Downloads")` ) and try running
the test case again.
> Implement NiFi test harness
> ---
>
> Key: NIFI-5318
> URL: https://issues.apache.org/jira/browse/NIFI-5318
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Peter Horvath
>Priority: Major
>
> Currently, it is not really possible to automatically test the behaviour of a
> specific NiFi flow and make unit test type asserts if it works as expected.
> For example, if the expected behaviour of a NiFi flow is that a file placed
> to a specific directory will trigger some operation after which some output
> file will appear at another directory, once currently can only do one thing:
> test the NiFi flow manually.
> Manual testing is especially hard to manage if a NiFi flow is being actively
> developed: any change to a complex, existing NiFi flow might require a lot of
> manual testing just to ensure there are no regressions introduced.
> Some kind of Java API that allows managing a NiFi instance and manipulating
> flow deployments like for example, [Codehaus
> Cargo|]https://codehaus-cargo.github.io/] would be of great help.
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2872: NIFI-5318 Implement NiFi test harness: initial commit of n...
Github user peter-gergely-horvath commented on the issue:
https://github.com/apache/nifi/pull/2872
@MikeThomsen This is kind of tricky: NiFi flows refer the precise version
of NiFi within the flow XML file, so you have something like this for the
processor definitions:
```
org.apache.nifi.processors.standard.GetHTTP
org.apache.nifi
nifi-standard-nar
1.7.1
```
When you export a flow file to test, I would expect the you to use
precisely the same NiFi version as the one used to export the flow file.
I think there is a little bit of confusion here: please note that these
tests do not test the test-harness. The test-harness itself is not tested (kind
of a chicken-and-egg issue): these test cases are merely _samples_, which are
referenced in the documentation: they demonstrate how an _end-user_ could
create test cases for his/her own flows.
Please download the ZIP of NiFi 1.7.1 to the `Downloads` directory within
your user home (referenced by
`org.apache.nifi.testharness.samples.Constants#NIFI_ZIP_DIR` as
`new File(System.getProperty("user.home"), "Downloads")` ) and try running
the test case again.
---
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop pulling data if the data contains invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665461#comment-16665461 ] Gardella Juan Pablo commented on NIFI-5761: --- Similar type of issue solved for Kafka processors. > ReplaceText processor can stop pulling data if the data contains invalid > expressions > > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0 > Environment: ALL >Reporter: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop pulling data if the data contains invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665460#comment-16665460 ] Gardella Juan Pablo commented on NIFI-5761: --- I will work on the patch. > ReplaceText processor can stop pulling data if the data contains invalid > expressions > > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0 > Environment: ALL >Reporter: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5761) ReplaceText processor can stop pulling data if the data contains invalid expressions
Gardella Juan Pablo created NIFI-5761: - Summary: ReplaceText processor can stop pulling data if the data contains invalid expressions Key: NIFI-5761 URL: https://issues.apache.org/jira/browse/NIFI-5761 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.5.0 Environment: ALL Reporter: Gardella Juan Pablo Given a flowfile body with nifi expression, when _ReplaceText_ processor evaluates it and the expression throws an exception, the processor will rollback the flowfile and keep trying to evaluate instead of send the flowfile to _failure_ relationshipt. Discussion Thread: http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5318) Implement NiFi test harness
[
https://issues.apache.org/jira/browse/NIFI-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665438#comment-16665438
]
ASF GitHub Bot commented on NIFI-5318:
--
Github user peter-gergely-horvath commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2872#discussion_r228606645
--- Diff:
nifi-testharness/src/test/java/org/apache/nifi/testharness/samples/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.nifi.testharness.samples;
+
+import java.io.File;
+
+public final class Constants {
+
+static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
+
+// NOTE: you will have to have the NiFi distribution ZIP placed into
this directory.
+// Its version must be the same as the one referenced in the flow.xml,
otherwise it will not work!
+static final File NIFI_ZIP_DIR = new
File(System.getProperty("user.home"), "Downloads");
--- End diff --
This will be relative to the `nifi_testharness_nifi_home` directory, from
which the test cases are executed. Since the test case should clear up that
directory automatically, I think this should be fine.
> Implement NiFi test harness
> ---
>
> Key: NIFI-5318
> URL: https://issues.apache.org/jira/browse/NIFI-5318
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Peter Horvath
>Priority: Major
>
> Currently, it is not really possible to automatically test the behaviour of a
> specific NiFi flow and make unit test type asserts if it works as expected.
> For example, if the expected behaviour of a NiFi flow is that a file placed
> to a specific directory will trigger some operation after which some output
> file will appear at another directory, once currently can only do one thing:
> test the NiFi flow manually.
> Manual testing is especially hard to manage if a NiFi flow is being actively
> developed: any change to a complex, existing NiFi flow might require a lot of
> manual testing just to ensure there are no regressions introduced.
> Some kind of Java API that allows managing a NiFi instance and manipulating
> flow deployments like for example, [Codehaus
> Cargo|]https://codehaus-cargo.github.io/] would be of great help.
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2872: NIFI-5318 Implement NiFi test harness: initial comm...
Github user peter-gergely-horvath commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2872#discussion_r228606645
--- Diff:
nifi-testharness/src/test/java/org/apache/nifi/testharness/samples/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.nifi.testharness.samples;
+
+import java.io.File;
+
+public final class Constants {
+
+static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
+
+// NOTE: you will have to have the NiFi distribution ZIP placed into
this directory.
+// Its version must be the same as the one referenced in the flow.xml,
otherwise it will not work!
+static final File NIFI_ZIP_DIR = new
File(System.getProperty("user.home"), "Downloads");
--- End diff --
This will be relative to the `nifi_testharness_nifi_home` directory, from
which the test cases are executed. Since the test case should clear up that
directory automatically, I think this should be fine.
---
[jira] [Created] (NIFI-5760) TeradataAdapter for GenerateTableFetch and QueryDatabaseTable
Siddharth created NIFI-5760: --- Summary: TeradataAdapter for GenerateTableFetch and QueryDatabaseTable Key: NIFI-5760 URL: https://issues.apache.org/jira/browse/NIFI-5760 Project: Apache NiFi Issue Type: New Feature Components: Configuration Reporter: Siddharth Fix For: 1.8.0 Hi, We came across a requirement to fetch huge amount of data from Teradata but since there are no adapter for teradata available we were forced to use MSSQL, It generated a query but it required some modification to make it teradata compatible. So it will be great if we can have a Teradata adapter for GenerateTableFetch, QUeryDatabase and other JDBC component -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-208) Export flow version from UI
Pierre Villard created NIFIREG-208: -- Summary: Export flow version from UI Key: NIFIREG-208 URL: https://issues.apache.org/jira/browse/NIFIREG-208 Project: NiFi Registry Issue Type: Improvement Reporter: Pierre Villard It'd be nice to add a link below or next to each flow version comment to let a user download the JSON representing the given version. Similarly, in addition to the 'Delete' action for a flow, there should be an 'Import' action to import a JSON as a new version of the flow. All the REST APIs are already available, it's purely a UI improvement for a better user experience (in addition to the CLI option). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFIREG-208) Export/import flow version from UI
[ https://issues.apache.org/jira/browse/NIFIREG-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFIREG-208: --- Summary: Export/import flow version from UI (was: Export flow version from UI) > Export/import flow version from UI > -- > > Key: NIFIREG-208 > URL: https://issues.apache.org/jira/browse/NIFIREG-208 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Pierre Villard >Priority: Major > > It'd be nice to add a link below or next to each flow version comment to let > a user download the JSON representing the given version. Similarly, in > addition to the 'Delete' action for a flow, there should be an 'Import' > action to import a JSON as a new version of the flow. All the REST APIs are > already available, it's purely a UI improvement for a better user experience > (in addition to the CLI option). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5759) Version Control Selections
Joseph Rosado created NIFI-5759: --- Summary: Version Control Selections Key: NIFI-5759 URL: https://issues.apache.org/jira/browse/NIFI-5759 Project: Apache NiFi Issue Type: Improvement Reporter: Joseph Rosado When using the Registry to version control process groups, I would like the option to choose which changes should be included in a change set. Currently, it's all or nothing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5758) Duplicate Variables
[ https://issues.apache.org/jira/browse/NIFI-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665326#comment-16665326 ] Pierre Villard commented on NIFI-5758: -- I don't think variables are duplicated, it's just that you can override the value of a variable defined in a process group in embedded process groups. You can have a look here: [https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#variable-scope] Let us know if you still think there is an issue. > Duplicate Variables > --- > > Key: NIFI-5758 > URL: https://issues.apache.org/jira/browse/NIFI-5758 > Project: Apache NiFi > Issue Type: Bug > Components: Variable Registry >Affects Versions: 1.7.1 >Reporter: Joseph Rosado >Priority: Major > Attachments: nifi-duplicate-variables.PNG > > > We have a process group with variables defined for the entire flow. However, > the variables are duplicated in each nested process group. Therefore, making > it difficult to manage, and identify issues. (see attached) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5758) Duplicate Variables
[ https://issues.apache.org/jira/browse/NIFI-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Rosado updated NIFI-5758: Description: We have a process group with variables defined for the entire flow. However, the variables are duplicated in each nested process group. Therefore, making it difficult to manage, and identify issues. (see attached) (was: We have a process group with variables defined for the entire flow. However, the variables are duplicated in each nested process group. Therefore, making it difficult to manage, and identify issues.) > Duplicate Variables > --- > > Key: NIFI-5758 > URL: https://issues.apache.org/jira/browse/NIFI-5758 > Project: Apache NiFi > Issue Type: Bug > Components: Variable Registry >Affects Versions: 1.7.1 >Reporter: Joseph Rosado >Priority: Major > Attachments: nifi-duplicate-variables.PNG > > > We have a process group with variables defined for the entire flow. However, > the variables are duplicated in each nested process group. Therefore, making > it difficult to manage, and identify issues. (see attached) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5758) Duplicate Variables
[ https://issues.apache.org/jira/browse/NIFI-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Rosado updated NIFI-5758: Attachment: nifi-duplicate-variables.PNG > Duplicate Variables > --- > > Key: NIFI-5758 > URL: https://issues.apache.org/jira/browse/NIFI-5758 > Project: Apache NiFi > Issue Type: Bug > Components: Variable Registry >Affects Versions: 1.7.1 >Reporter: Joseph Rosado >Priority: Major > Attachments: nifi-duplicate-variables.PNG > > > We have a process group with variables defined for the entire flow. However, > the variables are duplicated in each nested process group. Therefore, making > it difficult to manage, and identify issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5758) Duplicate Variables
Joseph Rosado created NIFI-5758: --- Summary: Duplicate Variables Key: NIFI-5758 URL: https://issues.apache.org/jira/browse/NIFI-5758 Project: Apache NiFi Issue Type: Bug Components: Variable Registry Affects Versions: 1.7.1 Reporter: Joseph Rosado Attachments: nifi-duplicate-variables.PNG We have a process group with variables defined for the entire flow. However, the variables are duplicated in each nested process group. Therefore, making it difficult to manage, and identify issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665278#comment-16665278 ] ASF GitHub Bot commented on NIFI-5224: -- Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/3041#discussion_r228563247 --- Diff: nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml --- @@ -147,6 +147,12 @@ 2.2.1 test + +org.apache.nifi +nifi-solr-client-api +1.8.0-SNAPSHOT +compile --- End diff -- Was able to try it out and you are right that it is working, but currently this is only because the service impl happens to be in the same NAR as the processors. If someone implemented their own SolrClientService in another NAR then it wouldn't be able to be used in the current state. If you just change the above dependency to provided and then change nifi-solr-nar/pom.xml where it has: ``` org.apache.nifi nifi-standard-services-api-nar 1.8.0-SNAPSHOT nar ``` To ``` org.apache.nifi nifi-solr-client-api-nar 1.8.0-SNAPSHOT nar ``` That should make it work correctly for both cases. > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3041: NIFI-5224 Added SolrClientService.
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/3041#discussion_r228563247 --- Diff: nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml --- @@ -147,6 +147,12 @@ 2.2.1 test + +org.apache.nifi +nifi-solr-client-api +1.8.0-SNAPSHOT +compile --- End diff -- Was able to try it out and you are right that it is working, but currently this is only because the service impl happens to be in the same NAR as the processors. If someone implemented their own SolrClientService in another NAR then it wouldn't be able to be used in the current state. If you just change the above dependency to provided and then change nifi-solr-nar/pom.xml where it has: ``` org.apache.nifi nifi-standard-services-api-nar 1.8.0-SNAPSHOT nar ``` To ``` org.apache.nifi nifi-solr-client-api-nar 1.8.0-SNAPSHOT nar ``` That should make it work correctly for both cases. ---
[jira] [Created] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
Arek Burdach created NIFI-5757:
--
Summary: AvroRecordSetWriter synchronize every access to
compiledAvroSchemaCache
Key: NIFI-5757
URL: https://issues.apache.org/jira/browse/NIFI-5757
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.7.1
Reporter: Arek Burdach
Avro record serialization is a quite expensive operation.
This stack trace I very often see in thread dumps:
{noformat}
Thread 48583: (state = BLOCKED)
- org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
@bci=9, line=124 (Compiled frame)
-
org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
@bci=96, line=92 (Compiled frame)
- sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
java.lang.Object[]) @bci=56 (Compiled frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=43 (Compiled frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
@bci=56, line=498 (Compiled frame)
-
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
frame)
- com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
@bci=24 (Compiled frame)
-
org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
org.apache.nifi.serialization.record.RecordSet,
org.apache.nifi.serialization.RecordSetWriterFactory,
org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
java.lang.String) @bci=71, line=169 (Compiled frame)
-
org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
@bci=94, line=412 (Compiled frame)
{noformat}
The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
every access to cache of compiled schemas.
I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
instead: https://github.com/apache/nifi/pull/3111
It is not a perfect fix because it removes cache size limitation which BTW was
hardcoded to {{20}}. Services can be reusable by many flows so such a hard
limit is not a good choice.
What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3111: AvroRecordSetWriter - Fix for slow synchronized acc...
GitHub user arkadius opened a pull request: https://github.com/apache/nifi/pull/3111 AvroRecordSetWriter - Fix for slow synchronized access to compiledAvroSchemaCache Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arkadius/nifi avro-synchronized-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3111 commit f35624187a7ccdb53171f3d143314da7a4e98455 Author: Arek Burdach Date: 2018-10-26T14:49:50Z Fix for slow synchronized access to compiledAvroSchemaCache in AvroRecordSetWriter. Used ConcurrentHashMap instead. ---
[GitHub] nifi pull request #3041: NIFI-5224 Added SolrClientService.
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/3041#discussion_r228542809 --- Diff: nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml --- @@ -147,6 +147,12 @@ 2.2.1 test + +org.apache.nifi +nifi-solr-client-api +1.8.0-SNAPSHOT +compile --- End diff -- This should be provided scope, using compile scope would mean the nifi-solr-client-api JAR would be packaged into two different NARs (the API NAR and the processors NAR) and at runtime it should only be in the API NAR. This relates to the other comment that the nifi-solr-nar pom needs to be updated to change its parent from standard services to the nifi-solr-client-api-nar. I'm not in a state to try it right now, but I would suspect that running a full build of this branch you would be able to create a SolrClientService, but not assign it in one of the Solr processors. ---
[jira] [Commented] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665233#comment-16665233 ] ASF GitHub Bot commented on NIFI-5224: -- Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi/pull/3041#discussion_r228542809 --- Diff: nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml --- @@ -147,6 +147,12 @@ 2.2.1 test + +org.apache.nifi +nifi-solr-client-api +1.8.0-SNAPSHOT +compile --- End diff -- This should be provided scope, using compile scope would mean the nifi-solr-client-api JAR would be packaged into two different NARs (the API NAR and the processors NAR) and at runtime it should only be in the API NAR. This relates to the other comment that the nifi-solr-nar pom needs to be updated to change its parent from standard services to the nifi-solr-client-api-nar. I'm not in a state to try it right now, but I would suspect that running a full build of this branch you would be able to create a SolrClientService, but not assign it in one of the Solr processors. > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5756) PutSQL Batching Fails when using DBCPConnectionPoolLookup
Joseph Rosado created NIFI-5756:
---
Summary: PutSQL Batching Fails when using DBCPConnectionPoolLookup
Key: NIFI-5756
URL: https://issues.apache.org/jira/browse/NIFI-5756
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.1
Reporter: Joseph Rosado
Attachments: flow-file-attribute.PNG, putsql-error.PNG,
putsql-properties.PNG
PutSQL processor fails to process flow files when batch size is greater than 1,
and using the DBCPConnectionPoolLookup controller service.
*{color:#262626}PutSQL[id=adc9c46f-0166-1000-5fb2-1a258f48216f] Failed to
process session due to Attributes must contain an attribute name
'database.name': org.apache.nifi.processor.exception.ProcessException:
Attributes must contain an attribute name 'database.name'{color}*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665216#comment-16665216 ] ASF GitHub Bot commented on NIFI-5224: -- Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/3041#discussion_r228537197 --- Diff: nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml --- @@ -147,6 +147,12 @@ 2.2.1 test + +org.apache.nifi +nifi-solr-client-api +1.8.0-SNAPSHOT +provided --- End diff -- > The solr-bundle NAR pom also needs to be updated to have a parent of the client API NAR. Could you explain? I added the client api nar to the assemble, rebuilt and was able to bring in a new client service just fine. > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3041: NIFI-5224 Added SolrClientService.
Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/3041#discussion_r228537197 --- Diff: nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/pom.xml --- @@ -147,6 +147,12 @@ 2.2.1 test + +org.apache.nifi +nifi-solr-client-api +1.8.0-SNAPSHOT +provided --- End diff -- > The solr-bundle NAR pom also needs to be updated to have a parent of the client API NAR. Could you explain? I added the client api nar to the assemble, rebuilt and was able to bring in a new client service just fine. ---
[jira] [Commented] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665201#comment-16665201
]
ASF GitHub Bot commented on NIFI-5752:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228534325
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
+if (verifier.verify(nodeId, sslSession)) {
+logger.debug("Authorizing Client to Load Balance data");
return;
--- End diff --
I think falling back to the hostname from the socket whenever there is not
an exact match (i.e., the wildcard matches but not an exact string comparison)
is fair. Originally, we used the hostname directly from the socket, but as Koji
mentioned in #3109 we changed that behavior. This was done because when you
look at Provenance data (and in logs), what you may see is something like a
RECEIVE event with a transit URI of nifi://s7302.r720.y8302.mydomain.com
because that's the FQDN but the user typically references this node as say
nifi-01.mydomain.com. If the UI shows the node as nifi-01.mydomain.com in the
cluster table, then it is best to show that in the Provenance and logs as well.
This is especially true in virtual environments, running in Docker or in a
publish Cloud/VM where often the hostname reported by socket.getInetAddress()
is very different than what we typically like to see.
Does that make sense?
Also @kotarot thank you for noticing this and submitting this contribution!
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[GitHub] nifi pull request #3110: NIFI-5752: Load balancing fails with wildcard certs
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228534325
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
+if (verifier.verify(nodeId, sslSession)) {
+logger.debug("Authorizing Client to Load Balance data");
return;
--- End diff --
I think falling back to the hostname from the socket whenever there is not
an exact match (i.e., the wildcard matches but not an exact string comparison)
is fair. Originally, we used the hostname directly from the socket, but as Koji
mentioned in #3109 we changed that behavior. This was done because when you
look at Provenance data (and in logs), what you may see is something like a
RECEIVE event with a transit URI of nifi://s7302.r720.y8302.mydomain.com
because that's the FQDN but the user typically references this node as say
nifi-01.mydomain.com. If the UI shows the node as nifi-01.mydomain.com in the
cluster table, then it is best to show that in the Provenance and logs as well.
This is especially true in virtual environments, running in Docker or in a
publish Cloud/VM where often the hostname reported by socket.getInetAddress()
is very different than what we typically like to see.
Does that make sense?
Also @kotarot thank you for noticing this and submitting this contribution!
---
[jira] [Resolved] (NIFI-5689) ReplaceText does not handle end of line correctly on buffer boundary
[ https://issues.apache.org/jira/browse/NIFI-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-5689. --- Resolution: Fixed > ReplaceText does not handle end of line correctly on buffer boundary > > > Key: NIFI-5689 > URL: https://issues.apache.org/jira/browse/NIFI-5689 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Sergei Zhirikov >Assignee: Otto Fowler >Priority: Minor > Attachments: Text_Parsing_Bug.xml > > > ReplaceText appears to misbehave under the following conditions: > * The input flow file contains text with Windows-style line endings (CR-LF). > * ReplaceText is configured to perform "Regex Replace" in "Line-by-Line" > mode. > * The "Maximum Buffer Size" is set to a value smaller than the whole file > content, > but large enough to fit any of the text lines in the file. > * A CR-LF pair of characters in one of the lines happens to be split across > two buffers, > that is CR is the last character in one buffer and LF is the first one in the > following one. > An example flow template is attached to illustrate the problem. > In the example, the regular expression is intended to remove white space at > the end of each line. It operates as expected in all lines except the third > one (containing "GHI"). That line satisfies the conditions described above. > As a result the CR character in the end of the line is removed, which does > not happen in other lines. > In some more complicated cases both CR and LF end up being removed, > effectively resulting in two lines being joined into one. Although, I haven't > managed to create a simple test case to reproduce that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5689) ReplaceText does not handle end of line correctly on buffer boundary
[ https://issues.apache.org/jira/browse/NIFI-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665172#comment-16665172 ] Otto Fowler commented on NIFI-5689: --- NIFI-5711 resolves this issue > ReplaceText does not handle end of line correctly on buffer boundary > > > Key: NIFI-5689 > URL: https://issues.apache.org/jira/browse/NIFI-5689 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Sergei Zhirikov >Assignee: Otto Fowler >Priority: Minor > Attachments: Text_Parsing_Bug.xml > > > ReplaceText appears to misbehave under the following conditions: > * The input flow file contains text with Windows-style line endings (CR-LF). > * ReplaceText is configured to perform "Regex Replace" in "Line-by-Line" > mode. > * The "Maximum Buffer Size" is set to a value smaller than the whole file > content, > but large enough to fit any of the text lines in the file. > * A CR-LF pair of characters in one of the lines happens to be split across > two buffers, > that is CR is the last character in one buffer and LF is the first one in the > following one. > An example flow template is attached to illustrate the problem. > In the example, the regular expression is intended to remove white space at > the end of each line. It operates as expected in all lines except the third > one (containing "GHI"). That line satisfies the conditions described above. > As a result the CR character in the end of the line is removed, which does > not happen in other lines. > In some more complicated cases both CR and LF end up being removed, > effectively resulting in two lines being joined into one. Although, I haven't > managed to create a simple test case to reproduce that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5642) QueryCassandra processor : output FlowFiles as soon fetch_size is reached
[
https://issues.apache.org/jira/browse/NIFI-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665154#comment-16665154
]
ASF GitHub Bot commented on NIFI-5642:
--
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r228522688
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -191,76 +203,110 @@ public void onScheduled(final ProcessContext
context) {
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+FlowFile inputFlowFile = null;
FlowFile fileToProcess = null;
+
+Map attributes = null;
+
if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
// If we have no FlowFile, and all incoming connections are
self-loops then we can continue on.
// However, if we have no FlowFile and we have connections
coming from other Processors, then
// we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
return;
}
+
+attributes = inputFlowFile.getAttributes();
}
final ComponentLog logger = getLogger();
-final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
-final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
+final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(inputFlowFile).getValue();
+final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(inputFlowFile).asTimePeriod(TimeUnit.MILLISECONDS);
final String outputFormat =
context.getProperty(OUTPUT_FORMAT).getValue();
-final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
+final long maxRowsPerFlowFile =
context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
+final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(inputFlowFile).getValue());
final StopWatch stopWatch = new StopWatch(true);
-if (fileToProcess == null) {
-fileToProcess = session.create();
+if(inputFlowFile != null){
+session.transfer(inputFlowFile, REL_ORIGINAL);
}
try {
// The documentation for the driver recommends the session
remain open the entire time the processor is running
// and states that it is thread-safe. This is why
connectionSession is not in a try-with-resources.
final Session connectionSession = cassandraSession.get();
-final ResultSetFuture queryFuture =
connectionSession.executeAsync(selectQuery);
+final ResultSet resultSet;
+
+if (queryTimeout > 0) {
+resultSet = connectionSession.execute(selectQuery,
queryTimeout, TimeUnit.MILLISECONDS);
+}else{
+resultSet = connectionSession.execute(selectQuery);
+}
+
final AtomicLong nrOfRows = new AtomicLong(0L);
-fileToProcess = session.write(fileToProcess, new
OutputStreamCallback() {
-@Override
-public void process(final OutputStream out) throws
IOException {
-try {
-logger.debug("Executing CQL query {}", new
Object[]{selectQuery});
-final ResultSet resultSet;
-if (queryTimeout > 0) {
-resultSet =
queryFuture.getUninterruptibly(queryTimeout, TimeUnit.MILLISECONDS);
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, queryTimeout,
TimeUnit.MILLISECONDS));
-} else if (JSON_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToJsonStream(resultSet, out, charset, queryTimeout,
TimeUnit.MILLISECONDS));
-}
-} else {
-
[jira] [Commented] (NIFI-5642) QueryCassandra processor : output FlowFiles as soon fetch_size is reached
[
https://issues.apache.org/jira/browse/NIFI-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665156#comment-16665156
]
ASF GitHub Bot commented on NIFI-5642:
--
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r228522827
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -400,77 +478,87 @@ public static long convertToJsonStream(final
ResultSet rs, final OutputStream ou
outStream.write("{\"results\":[".getBytes(charset));
final ColumnDefinitions columnDefinitions =
rs.getColumnDefinitions();
long nrOfRows = 0;
+long rowsAvailableWithoutFetching =
rs.getAvailableWithoutFetching();
+
if (columnDefinitions != null) {
-do {
-
-// Grab the ones we have
-int rowsAvailableWithoutFetching =
rs.getAvailableWithoutFetching();
-if (rowsAvailableWithoutFetching == 0) {
-// Get more
-if (timeout <= 0 || timeUnit == null) {
-rs.fetchMoreResults().get();
-} else {
-rs.fetchMoreResults().get(timeout, timeUnit);
-}
+
+// Grab the ones we have
+if (rowsAvailableWithoutFetching == 0) {
+// Get more
+if (timeout <= 0 || timeUnit == null) {
+rs.fetchMoreResults().get();
+} else {
+rs.fetchMoreResults().get(timeout, timeUnit);
}
+rowsAvailableWithoutFetching =
rs.getAvailableWithoutFetching();
+}
-for (Row row : rs) {
-if (nrOfRows != 0) {
+if(maxRowsPerFlowFile == 0){
+maxRowsPerFlowFile = rowsAvailableWithoutFetching;
+}
+Row row;
+while(nrOfRows < maxRowsPerFlowFile){
+try {
+row = rs.iterator().next();
+}catch (NoSuchElementException nsee){
+//nrOfRows -= 1;
--- End diff --
True!
> QueryCassandra processor : output FlowFiles as soon fetch_size is reached
> -
>
> Key: NIFI-5642
> URL: https://issues.apache.org/jira/browse/NIFI-5642
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: André Gomes Lamas Otero
>Priority: Major
>
> When I'm using QueryCassandra alongside with fetch_size parameter I expected
> that as soon my reader reaches the fetch_size the processor outputs some data
> to be processed by the next processor, but QueryCassandra reads all the data,
> then output the flow files.
> I'll start to work on a patch for this situation, I'll appreciate any
> suggestion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3051: NIFI-5642: QueryCassandra processor : output FlowFi...
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r228522688
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -191,76 +203,110 @@ public void onScheduled(final ProcessContext
context) {
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+FlowFile inputFlowFile = null;
FlowFile fileToProcess = null;
+
+Map attributes = null;
+
if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
// If we have no FlowFile, and all incoming connections are
self-loops then we can continue on.
// However, if we have no FlowFile and we have connections
coming from other Processors, then
// we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
return;
}
+
+attributes = inputFlowFile.getAttributes();
}
final ComponentLog logger = getLogger();
-final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
-final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
+final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(inputFlowFile).getValue();
+final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(inputFlowFile).asTimePeriod(TimeUnit.MILLISECONDS);
final String outputFormat =
context.getProperty(OUTPUT_FORMAT).getValue();
-final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
+final long maxRowsPerFlowFile =
context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
+final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(inputFlowFile).getValue());
final StopWatch stopWatch = new StopWatch(true);
-if (fileToProcess == null) {
-fileToProcess = session.create();
+if(inputFlowFile != null){
+session.transfer(inputFlowFile, REL_ORIGINAL);
}
try {
// The documentation for the driver recommends the session
remain open the entire time the processor is running
// and states that it is thread-safe. This is why
connectionSession is not in a try-with-resources.
final Session connectionSession = cassandraSession.get();
-final ResultSetFuture queryFuture =
connectionSession.executeAsync(selectQuery);
+final ResultSet resultSet;
+
+if (queryTimeout > 0) {
+resultSet = connectionSession.execute(selectQuery,
queryTimeout, TimeUnit.MILLISECONDS);
+}else{
+resultSet = connectionSession.execute(selectQuery);
+}
+
final AtomicLong nrOfRows = new AtomicLong(0L);
-fileToProcess = session.write(fileToProcess, new
OutputStreamCallback() {
-@Override
-public void process(final OutputStream out) throws
IOException {
-try {
-logger.debug("Executing CQL query {}", new
Object[]{selectQuery});
-final ResultSet resultSet;
-if (queryTimeout > 0) {
-resultSet =
queryFuture.getUninterruptibly(queryTimeout, TimeUnit.MILLISECONDS);
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, queryTimeout,
TimeUnit.MILLISECONDS));
-} else if (JSON_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToJsonStream(resultSet, out, charset, queryTimeout,
TimeUnit.MILLISECONDS));
-}
-} else {
-resultSet = queryFuture.getUninterruptibly();
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, 0, null));
-}
[GitHub] nifi pull request #3051: NIFI-5642: QueryCassandra processor : output FlowFi...
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r228522827
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -400,77 +478,87 @@ public static long convertToJsonStream(final
ResultSet rs, final OutputStream ou
outStream.write("{\"results\":[".getBytes(charset));
final ColumnDefinitions columnDefinitions =
rs.getColumnDefinitions();
long nrOfRows = 0;
+long rowsAvailableWithoutFetching =
rs.getAvailableWithoutFetching();
+
if (columnDefinitions != null) {
-do {
-
-// Grab the ones we have
-int rowsAvailableWithoutFetching =
rs.getAvailableWithoutFetching();
-if (rowsAvailableWithoutFetching == 0) {
-// Get more
-if (timeout <= 0 || timeUnit == null) {
-rs.fetchMoreResults().get();
-} else {
-rs.fetchMoreResults().get(timeout, timeUnit);
-}
+
+// Grab the ones we have
+if (rowsAvailableWithoutFetching == 0) {
+// Get more
+if (timeout <= 0 || timeUnit == null) {
+rs.fetchMoreResults().get();
+} else {
+rs.fetchMoreResults().get(timeout, timeUnit);
}
+rowsAvailableWithoutFetching =
rs.getAvailableWithoutFetching();
+}
-for (Row row : rs) {
-if (nrOfRows != 0) {
+if(maxRowsPerFlowFile == 0){
+maxRowsPerFlowFile = rowsAvailableWithoutFetching;
+}
+Row row;
+while(nrOfRows < maxRowsPerFlowFile){
+try {
+row = rs.iterator().next();
+}catch (NoSuchElementException nsee){
+//nrOfRows -= 1;
--- End diff --
True!
---
[jira] [Updated] (NIFI-5726) Update README with Slack Workspace information
[ https://issues.apache.org/jira/browse/NIFI-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-5726: --- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Update README with Slack Workspace information > -- > > Key: NIFI-5726 > URL: https://issues.apache.org/jira/browse/NIFI-5726 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.9.0 > > > Update README with Slack Workspace information -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5726) Update README with Slack Workspace information
[ https://issues.apache.org/jira/browse/NIFI-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665139#comment-16665139 ] ASF GitHub Bot commented on NIFI-5726: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3093 > Update README with Slack Workspace information > -- > > Key: NIFI-5726 > URL: https://issues.apache.org/jira/browse/NIFI-5726 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.9.0 > > > Update README with Slack Workspace information -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3093: NIFI-5726 - Update README with Slack Workspace info...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3093 ---
[jira] [Commented] (NIFI-5726) Update README with Slack Workspace information
[ https://issues.apache.org/jira/browse/NIFI-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665138#comment-16665138 ] ASF subversion and git services commented on NIFI-5726: --- Commit e4b702ed4f1e144f8ad667789ac3cddc981803f8 in nifi's branch refs/heads/master from [~pvillard] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e4b702e ] NIFI-5726 - Update README with Slack Workspace information This closes #3093 Signed-off-by: Mike Thomsen > Update README with Slack Workspace information > -- > > Key: NIFI-5726 > URL: https://issues.apache.org/jira/browse/NIFI-5726 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > Update README with Slack Workspace information -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5318) Implement NiFi test harness
[
https://issues.apache.org/jira/browse/NIFI-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665124#comment-16665124
]
ASF GitHub Bot commented on NIFI-5318:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2872#discussion_r228511342
--- Diff:
nifi-testharness/src/test/java/org/apache/nifi/testharness/samples/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.nifi.testharness.samples;
+
+import java.io.File;
+
+public final class Constants {
+
+static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
+
+// NOTE: you will have to have the NiFi distribution ZIP placed into
this directory.
+// Its version must be the same as the one referenced in the flow.xml,
otherwise it will not work!
+static final File NIFI_ZIP_DIR = new
File(System.getProperty("user.home"), "Downloads");
--- End diff --
Since this is an internal unit test, it should be pointing to the build
output of nifi-assembly.
> Implement NiFi test harness
> ---
>
> Key: NIFI-5318
> URL: https://issues.apache.org/jira/browse/NIFI-5318
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Peter Horvath
>Priority: Major
>
> Currently, it is not really possible to automatically test the behaviour of a
> specific NiFi flow and make unit test type asserts if it works as expected.
> For example, if the expected behaviour of a NiFi flow is that a file placed
> to a specific directory will trigger some operation after which some output
> file will appear at another directory, once currently can only do one thing:
> test the NiFi flow manually.
> Manual testing is especially hard to manage if a NiFi flow is being actively
> developed: any change to a complex, existing NiFi flow might require a lot of
> manual testing just to ensure there are no regressions introduced.
> Some kind of Java API that allows managing a NiFi instance and manipulating
> flow deployments like for example, [Codehaus
> Cargo|]https://codehaus-cargo.github.io/] would be of great help.
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2872: NIFI-5318 Implement NiFi test harness: initial comm...
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2872#discussion_r228511342
--- Diff:
nifi-testharness/src/test/java/org/apache/nifi/testharness/samples/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.nifi.testharness.samples;
+
+import java.io.File;
+
+public final class Constants {
+
+static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
+
+// NOTE: you will have to have the NiFi distribution ZIP placed into
this directory.
+// Its version must be the same as the one referenced in the flow.xml,
otherwise it will not work!
+static final File NIFI_ZIP_DIR = new
File(System.getProperty("user.home"), "Downloads");
--- End diff --
Since this is an internal unit test, it should be pointing to the build
output of nifi-assembly.
---
[jira] [Commented] (NIFI-5224) Add SolrClientService
[
https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665111#comment-16665111
]
ASF GitHub Bot commented on NIFI-5224:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3041#discussion_r228507330
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/SolrProcessor.java
---
@@ -176,70 +172,11 @@ protected final KeytabUser getKerberosKeytabUser() {
final protected Collection
customValidate(ValidationContext context) {
final List problems = new ArrayList<>();
-if
(SOLR_TYPE_CLOUD.equals(context.getProperty(SOLR_TYPE).getValue())) {
-final String collection =
context.getProperty(COLLECTION).getValue();
-if (collection == null || collection.trim().isEmpty()) {
-problems.add(new ValidationResult.Builder()
-.subject(COLLECTION.getName())
-.input(collection).valid(false)
-.explanation("A collection must specified for Solr
Type of Cloud")
-.build());
-}
-}
-
-// For solr cloud the location will be the ZooKeeper host:port so
we can't validate the SSLContext, but for standard solr
-// we can validate if the url starts with https we need an
SSLContextService, if it starts with http we can't have an SSLContextService
-if
(SOLR_TYPE_STANDARD.equals(context.getProperty(SOLR_TYPE).getValue())) {
-final String solrLocation =
context.getProperty(SOLR_LOCATION).evaluateAttributeExpressions().getValue();
-if (solrLocation != null) {
-final SSLContextService sslContextService =
context.getProperty(SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
-if (solrLocation.startsWith("https:") && sslContextService
== null) {
-problems.add(new ValidationResult.Builder()
-.subject(SSL_CONTEXT_SERVICE.getDisplayName())
-.valid(false)
-.explanation("an SSLContextService must be
provided when using https")
-.build());
-} else if (solrLocation.startsWith("http:") &&
sslContextService != null) {
-problems.add(new ValidationResult.Builder()
-.subject(SSL_CONTEXT_SERVICE.getDisplayName())
-.valid(false)
-.explanation("an SSLContextService can not be
provided when using http")
-.build());
-}
-}
-}
-
-// Validate that we username and password are provided together,
or that neither are provided
-final String username =
context.getProperty(BASIC_USERNAME).evaluateAttributeExpressions().getValue();
-final String password =
context.getProperty(BASIC_PASSWORD).evaluateAttributeExpressions().getValue();
-
-final boolean basicUsernameProvided =
!StringUtils.isBlank(username);
-final boolean basicPasswordProvided =
!StringUtils.isBlank(password);
-
-if (basicUsernameProvided && !basicPasswordProvided) {
-problems.add(new ValidationResult.Builder()
-.subject(BASIC_PASSWORD.getDisplayName())
-.valid(false)
-.explanation("a password must be provided for the
given username")
-.build());
-}
-
-if (basicPasswordProvided && !basicUsernameProvided) {
-problems.add(new ValidationResult.Builder()
-.subject(BASIC_USERNAME.getDisplayName())
-.valid(false)
-.explanation("a username must be provided for the
given password")
-.build());
-}
-
-// Validate that only kerberos or basic auth can be set, but not
both
-final KerberosCredentialsService kerberosCredentialsService =
context.getProperty(KERBEROS_CREDENTIALS_SERVICE).asControllerService(KerberosCredentialsService.class);
-if (kerberosCredentialsService != null && basicUsernameProvided &&
basicPasswordProvided) {
-problems.add(new ValidationResult.Builder()
-.subject(KERBEROS_CREDENTIALS_SERVICE.getDisplayName())
-.valid(false)
-.explanation("basic auth and kerberos cannot be
configured at the same time")
-.build());
+
[GitHub] nifi pull request #3041: NIFI-5224 Added SolrClientService.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3041#discussion_r228507330
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/SolrProcessor.java
---
@@ -176,70 +172,11 @@ protected final KeytabUser getKerberosKeytabUser() {
final protected Collection
customValidate(ValidationContext context) {
final List problems = new ArrayList<>();
-if
(SOLR_TYPE_CLOUD.equals(context.getProperty(SOLR_TYPE).getValue())) {
-final String collection =
context.getProperty(COLLECTION).getValue();
-if (collection == null || collection.trim().isEmpty()) {
-problems.add(new ValidationResult.Builder()
-.subject(COLLECTION.getName())
-.input(collection).valid(false)
-.explanation("A collection must specified for Solr
Type of Cloud")
-.build());
-}
-}
-
-// For solr cloud the location will be the ZooKeeper host:port so
we can't validate the SSLContext, but for standard solr
-// we can validate if the url starts with https we need an
SSLContextService, if it starts with http we can't have an SSLContextService
-if
(SOLR_TYPE_STANDARD.equals(context.getProperty(SOLR_TYPE).getValue())) {
-final String solrLocation =
context.getProperty(SOLR_LOCATION).evaluateAttributeExpressions().getValue();
-if (solrLocation != null) {
-final SSLContextService sslContextService =
context.getProperty(SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
-if (solrLocation.startsWith("https:") && sslContextService
== null) {
-problems.add(new ValidationResult.Builder()
-.subject(SSL_CONTEXT_SERVICE.getDisplayName())
-.valid(false)
-.explanation("an SSLContextService must be
provided when using https")
-.build());
-} else if (solrLocation.startsWith("http:") &&
sslContextService != null) {
-problems.add(new ValidationResult.Builder()
-.subject(SSL_CONTEXT_SERVICE.getDisplayName())
-.valid(false)
-.explanation("an SSLContextService can not be
provided when using http")
-.build());
-}
-}
-}
-
-// Validate that we username and password are provided together,
or that neither are provided
-final String username =
context.getProperty(BASIC_USERNAME).evaluateAttributeExpressions().getValue();
-final String password =
context.getProperty(BASIC_PASSWORD).evaluateAttributeExpressions().getValue();
-
-final boolean basicUsernameProvided =
!StringUtils.isBlank(username);
-final boolean basicPasswordProvided =
!StringUtils.isBlank(password);
-
-if (basicUsernameProvided && !basicPasswordProvided) {
-problems.add(new ValidationResult.Builder()
-.subject(BASIC_PASSWORD.getDisplayName())
-.valid(false)
-.explanation("a password must be provided for the
given username")
-.build());
-}
-
-if (basicPasswordProvided && !basicUsernameProvided) {
-problems.add(new ValidationResult.Builder()
-.subject(BASIC_USERNAME.getDisplayName())
-.valid(false)
-.explanation("a username must be provided for the
given password")
-.build());
-}
-
-// Validate that only kerberos or basic auth can be set, but not
both
-final KerberosCredentialsService kerberosCredentialsService =
context.getProperty(KERBEROS_CREDENTIALS_SERVICE).asControllerService(KerberosCredentialsService.class);
-if (kerberosCredentialsService != null && basicUsernameProvided &&
basicPasswordProvided) {
-problems.add(new ValidationResult.Builder()
-.subject(KERBEROS_CREDENTIALS_SERVICE.getDisplayName())
-.valid(false)
-.explanation("basic auth and kerberos cannot be
configured at the same time")
-.build());
+List _temp = new
ArrayList<>(validateConnectionDetails(context));
+if (_temp.size() == 0 &&
context.getProperty(CLIENT_SERVICE).isSet()) {
--- End diff --
Fixed that too.
---
[jira] [Assigned] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[ https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda reassigned MINIFICPP-648: Assignee: Arpad Boda (was: Mr TheSegfault) > add processor and add processor with linkage nomenclature is confusing > -- > > Key: MINIFICPP-648 > URL: https://issues.apache.org/jira/browse/MINIFICPP-648 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Arpad Boda >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > > add_processor should be changed to always add a processor with linkage > without compelling documentation as why this exists.. As a result we will > need to add a create_processor function to create one without adding it to > the flow ( certain use cases where a flow isn't needed such as invokehttp or > listenhttp ) this can be moved to 0.7.0 if we tag before recent commits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5747:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664849#comment-16664849
]
ASF subversion and git services commented on NIFI-5747:
---
Commit fdbcf34281bf6ee71f4f5ba6b9445254bc874752 in nifi's branch
refs/heads/master from juliansniff
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=fdbcf34 ]
NIFI-5747 fix NPE when redirecting from HTTPS to HTTP for InvokeHTTP
Signed-off-by: Koji Kawamura
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664847#comment-16664847
]
Koji Kawamura commented on NIFI-5747:
-
The patch looks good. Confirmed the issue has been addressed. Merging to
master. Thanks again, [~juliansniff]!
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5747:
Affects Version/s: 1.0.0
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5747:
Fix Version/s: 1.9.0
Component/s: Extensions
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5675) Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent expressions
[
https://issues.apache.org/jira/browse/NIFI-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kotaro Terada reassigned NIFI-5675:
---
Assignee: Kotaro Terada
> Some tests in ConvertExcelToCSVProcessorTest fail because of locale-dependent
> expressions
> -
>
> Key: NIFI-5675
> URL: https://issues.apache.org/jira/browse/NIFI-5675
> Project: Apache NiFi
> Issue Type: Test
> Components: Extensions
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Some tests in {{ConvertExcelToCSVProcessorTest}} fail in some locales because
> text in the assertions is locale-dependent.
> The problem is that DateTime expressions (such as "Sunday, January 01, 2017")
> are based on English and locale-dependent. When the locale of JVM is "ja_JP"
> for example, the tests fail as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR] Tests run: 13, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 5.32
> s <<< FAILURE! - in
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest
> [ERROR]
> testSkipRows(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.146 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRows(ConvertExcelToCSVProcessorTest.java:153)
> [ERROR]
> testCustomDelimiters(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.03 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...5
> 1234.46|12:00:00 [PM|£ 123.45
> 1234.5|Sunday, January 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 PM]|£ 1,023.45
> 9.88E...> but was:<...5
> 1234.46|12:00:00 [午後|£ 123.45
> 1234.5|日曜日, 1月 01, 2017|¥ 123.45
> 1,234.46|1/1/17 12:00|$ 1,023.45
> 1,234.4560|12:00 午後]|£ 1,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testCustomDelimiters(ConvertExcelToCSVProcessorTest.java:266)
> [ERROR]
> testQuoting(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.029 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...45
> 1234.46,12:00:00 [PM,£ 123.45
> 1234.5,"Sunday, January 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 PM],"£ 1,023.45"
> 9.88...> but was:<...45
> 1234.46,12:00:00 [午後,£ 123.45
> 1234.5,"日曜日, 1月 01, 2017",¥ 123.45
> "1,234.46",1/1/17 12:00,"$ 1,023.45"
> "1,234.4560",12:00 午後],"£ 1,023.45"
> 9.88...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testQuoting(ConvertExcelToCSVProcessorTest.java:125)
> [ERROR]
> testSkipRowsWithEL(org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest)
> Time elapsed: 0.02 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<1234.46,12:00:00 [PM,£ 123.45
> 1234.5,Sunday\, January 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 PM],£ 1\,023.45
> 9.88E...> but was:<1234.46,12:00:00 [午後,£ 123.45
> 1234.5,日曜日\, 1月 01\, 2017,¥ 123.45
> 1\,234.46,1/1/17 12:00,$ 1\,023.45
> 1\,234.4560,12:00 午後],£ 1\,023.45
> 9.88E...>
> at
> org.apache.nifi.processors.poi.ConvertExcelToCSVProcessorTest.testSkipRowsWithEL(ConvertExcelToCSVProcessorTest.java:181)
> {noformat}
> To fix this, we can convert the expected DateTime strings using
> `DateTimeFormatter` in the assertions.
> Note that similar issues and fixes (for a different class) are here:
> - https://issues.apache.org/jira/browse/NIFI-2683
> - https://issues.apache.org/jira/browse/NIFI-3483
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kotaro Terada updated NIFI-5752:
Status: Patch Available (was: Open)
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This problem occurs because in {{authorize}} method in
> {{ClusterLoadBalanceAuthorizer}} class, authorization is tried by just
> matching strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5681) A test in TestVersionedFlowSnapshotMetadataResult fails because it depends on locale
[
https://issues.apache.org/jira/browse/NIFI-5681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kotaro Terada reassigned NIFI-5681:
---
Assignee: Kotaro Terada
> A test in TestVersionedFlowSnapshotMetadataResult fails because it depends on
> locale
>
>
> Key: NIFI-5681
> URL: https://issues.apache.org/jira/browse/NIFI-5681
> Project: Apache NiFi
> Issue Type: Test
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Test {{testWriteSimpleVersionedFlowSnapshotResult}} in
> {{TestVersionedFlowSnapshotMetadataResult}} class fails with an assertion
> error when the locale is not "en_US" (or similar locale). When the locale is
> "ja_JP", the test fails as follows:
> {noformat}
> [INFO] Running
> org.apache.nifi.toolkit.cli.impl.result.TestVersionedFlowSnapshotMetadataResult
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.028
> s <<< FAILURE! - in
> org.apache.nifi.toolkit.cli.impl.result.TestVersionedFlowSnapshotMetadataResult
> [ERROR]
> testWriteSimpleVersionedFlowSnapshotResult(org.apache.nifi.toolkit.cli.impl.result.TestVersionedFlowSnapshotMetadataResult)
> Time elapsed: 0.027 s <<< FAILURE!
> java.lang.AssertionError
> at
> org.apache.nifi.toolkit.cli.impl.result.TestVersionedFlowSnapshotMetadataResult.testWriteSimpleVersionedFlowSnapshotResult(TestVersionedFlowSnapshotMetadataResult.java:79)
> {noformat}
> The content of {{resultOut}} is now:
> {noformat}
> Ver Date Author Message
>
> --- -- --
>
> 1 水, 2 14 2018 12:00 JST user1This is a long comment, longer than
> t...
> 2 水, 2 14 2018 12:30 JST user2This is v2
>
> {noformat}
> Even if checking the heading two lines (as we do now), length of a line is
> different by each locale because width of columns depends on locale. We can
> deal with this problem using regex.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5676) A test in PutORCTest fails because it depends on timezone
[
https://issues.apache.org/jira/browse/NIFI-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kotaro Terada reassigned NIFI-5676:
---
Assignee: Kotaro Terada
> A test in PutORCTest fails because it depends on timezone
> -
>
> Key: NIFI-5676
> URL: https://issues.apache.org/jira/browse/NIFI-5676
> Project: Apache NiFi
> Issue Type: Test
> Components: Extensions
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Fix For: 1.8.0
>
>
> Test {{testWriteORCWithAvroLogicalTypes}} fails because it depends on
> timezone. It seems that, in this test, the converted date in ORC is "1 day"
> before the input date when the timezone is ahead of GMT. For example, when
> the timezone is "GMT+9", this test fails as follows:
> {noformat}
> [INFO] Running org.apache.nifi.processors.orc.PutORCTest
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.694
> s <<< FAILURE! - in org.apache.nifi.processors.orc.PutORCTest
> [ERROR]
> testWriteORCWithAvroLogicalTypes(org.apache.nifi.processors.orc.PutORCTest)
> Time elapsed: 0.146 s <<< FAILURE!
> java.lang.AssertionError: expected:<17813> but was:<17812>
> at
> org.apache.nifi.processors.orc.PutORCTest.lambda$testWriteORCWithAvroLogicalTypes$1(PutORCTest.java:250)
> at
> org.apache.nifi.processors.orc.PutORCTest.verifyORCUsers(PutORCTest.java:408)
> at
> org.apache.nifi.processors.orc.PutORCTest.testWriteORCWithAvroLogicalTypes(PutORCTest.java:246)
> {noformat}
> To fix this, we can set the timezone information in the assertion as we did
> in [NIFI-3718|https://issues.apache.org/jira/browse/NIFI-3718].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kotaro Terada reassigned NIFI-5752:
---
Assignee: Kotaro Terada
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This problem occurs because in {{authorize}} method in
> {{ClusterLoadBalanceAuthorizer}} class, authorization is tried by just
> matching strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664805#comment-16664805
]
Koji Kawamura commented on NIFI-5747:
-
[~juliansniff] Thanks for sending the patch! I've assigned you to this JIRA.
I'll review the patch.
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura reassigned NIFI-5747:
---
Assignee: Julian Sniffen
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julian Sniffen
>Assignee: Julian Sniffen
>Priority: Major
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664794#comment-16664794
]
ASF GitHub Bot commented on NIFI-5752:
--
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428335
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
+if (verifier.verify(nodeId, sslSession)) {
+logger.debug("Authorizing Client to Load Balance data");
--- End diff --
I agree with your idea. I'll fix it so!
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This problem occurs because in {{authorize}} method in
> {{ClusterLoadBalanceAuthorizer}} class, authorization is tried by just
> matching strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3110: NIFI-5752: Load balancing fails with wildcard certs
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428335
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
+if (verifier.verify(nodeId, sslSession)) {
+logger.debug("Authorizing Client to Load Balance data");
--- End diff --
I agree with your idea. I'll fix it so!
---
[GitHub] nifi pull request #3110: NIFI-5752: Load balancing fails with wildcard certs
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428297
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
--- End diff --
Good point. Instantiating in every iteration is wasteful. I'll fix it!
---
[jira] [Commented] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664792#comment-16664792
]
ASF GitHub Bot commented on NIFI-5752:
--
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428134
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
--- End diff --
In the existing code, we needed to search a target element in `Set
nodeIds` by the `contains` method. However, in this change, `nodeIds` is just
iterated in a loop, so it is reasonable to change it `List`.
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This problem occurs because in {{authorize}} method in
> {{ClusterLoadBalanceAuthorizer}} class, authorization is tried by just
> matching strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3110: NIFI-5752: Load balancing fails with wildcard certs
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428588
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
+if (verifier.verify(nodeId, sslSession)) {
+logger.debug("Authorizing Client to Load Balance data");
return;
--- End diff --
In my opinion, we just need to use HostnameVerifier to verify and use the
hostname derived from the socket. The reason is that, anyway, HostnameVerifier
could simply authorize a node using certs w/ or w/o wildcard, and I think the
hostname derived from the socket is enough. If there are cases where the
hostname derived from the socket and the hostname from Certificate Identities
are different, please ignore my option.
I'd also like to hear comment from @markap14 . Thank you.
---
[jira] [Commented] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664793#comment-16664793
]
ASF GitHub Bot commented on NIFI-5752:
--
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428297
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
--- End diff --
Good point. Instantiating in every iteration is wasteful. I'll fix it!
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This problem occurs because in {{authorize}} method in
> {{ClusterLoadBalanceAuthorizer}} class, authorization is tried by just
> matching strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5752) Load balancing fails with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664791#comment-16664791
]
ASF GitHub Bot commented on NIFI-5752:
--
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428588
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
-for (final String clientId : clientIdentities) {
-if (nodeIds.contains(clientId)) {
-logger.debug("Client ID '{}' is in the list of Nodes in
the Cluster. Authorizing Client to Load Balance data", clientId);
+for (final String nodeId : nodeIds) {
+final HostnameVerifier verifier = new
DefaultHostnameVerifier();
+if (verifier.verify(nodeId, sslSession)) {
+logger.debug("Authorizing Client to Load Balance data");
return;
--- End diff --
In my opinion, we just need to use HostnameVerifier to verify and use the
hostname derived from the socket. The reason is that, anyway, HostnameVerifier
could simply authorize a node using certs w/ or w/o wildcard, and I think the
hostname derived from the socket is enough. If there are cases where the
hostname derived from the socket and the hostname from Certificate Identities
are different, please ignore my option.
I'd also like to hear comment from @markap14 . Thank you.
> Load balancing fails with wildcard certs
>
>
> Key: NIFI-5752
> URL: https://issues.apache.org/jira/browse/NIFI-5752
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Priority: Major
>
> Load balancing fails when we construct a secure cluster with wildcard certs.
> For example, assume that we have a valid wildcard cert for {{*.example.com}}
> and a cluster consists of {{nf1.example.com}}, {{nf2.example.com}}, and
> {{nf3.example.com}} . We cannot transfer a FlowFile between nodes for load
> balancing because of the following authorization error:
> {noformat}
> 2018-10-25 19:05:13,520 WARN [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ClusterLoadBalanceAuthorizer Authorization failed for Client
> ID's [*.example.com] to Load Balance data because none of the ID's are known
> Cluster Node Identifiers
> 2018-10-25 19:05:13,521 ERROR [Load Balance Server Thread-2]
> o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
> /xxx.xxx.xxx.xxx:x
> org.apache.nifi.controller.queue.clustered.server.NotAuthorizedException:
> Client ID's [*.example.com] are not authorized to Load Balance data
> at
> org.apache.nifi.controller.queue.clustered.server.ClusterLoadBalanceAuthorizer.authorize(ClusterLoadBalanceAuthorizer.java:65)
> at
> org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:142)
> at
> org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This
[GitHub] nifi pull request #3110: NIFI-5752: Load balancing fails with wildcard certs
Github user kotarot commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3110#discussion_r228428134
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ClusterLoadBalanceAuthorizer.java
---
@@ -40,28 +42,23 @@ public ClusterLoadBalanceAuthorizer(final
ClusterCoordinator clusterCoordinator,
}
@Override
-public void authorize(final Collection clientIdentities)
throws NotAuthorizedException {
-if (clientIdentities == null) {
-logger.debug("Client Identities is null, so assuming that Load
Balancing communications are not secure. Authorizing client to participate in
Load Balancing");
-return;
-}
-
-final Set nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
+public void authorize(final SSLSession sslSession) throws
NotAuthorizedException {
+final List nodeIds =
clusterCoordinator.getNodeIdentifiers().stream()
.map(NodeIdentifier::getApiAddress)
-.collect(Collectors.toSet());
+.collect(Collectors.toList());
--- End diff --
In the existing code, we needed to search a target element in `Set
nodeIds` by the `contains` method. However, in this change, `nodeIds` is just
iterated in a loop, so it is reasonable to change it `List`.
---
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664751#comment-16664751 ] ASF GitHub Bot commented on NIFI-5745: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3108 > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-5745: Resolution: Fixed Status: Resolved (was: Patch Available) > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3108: NIFI-5745: When determining if backpressure should ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3108 ---
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664749#comment-16664749 ] ASF subversion and git services commented on NIFI-5745: --- Commit 234ddb0fe1a36ad947c340114058d82c777d791f in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=234ddb0 ] NIFI-5745: When determining if backpressure should be applied across nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue This closes #3108. Signed-off-by: Koji Kawamura > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664746#comment-16664746 ] ASF GitHub Bot commented on NIFI-5745: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3108 I tested the change with different partition strategies. Works properly without being live-locked. And back-pressure can be propagated to upstream when it should. LGTM, +1, merging. Thanks @markap14 ! > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3108: NIFI-5745: When determining if backpressure should be appl...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3108 I tested the change with different partition strategies. Works properly without being live-locked. And back-pressure can be propagated to upstream when it should. LGTM, +1, merging. Thanks @markap14 ! ---
