[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662556#comment-16662556 ] ASF GitHub Bot commented on NIFI-5745: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3108 NIFI-5745: When determining if backpressure should be applied across … …nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5745 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3108 commit 1ee39847e6ecc0860bb1e272a4aba0ebf23bde16 Author: Mark Payne Date: 2018-10-24T17:06:57Z NIFI-5745: When determining if backpressure should be applied across nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664711#comment-16664711 ] ASF GitHub Bot commented on NIFI-5745: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3108 Reviewing... I was able to reproduce the issue.  > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664746#comment-16664746 ] ASF GitHub Bot commented on NIFI-5745: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3108 I tested the change with different partition strategies. Works properly without being live-locked. And back-pressure can be propagated to upstream when it should. LGTM, +1, merging. Thanks @markap14 ! > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664749#comment-16664749 ] ASF subversion and git services commented on NIFI-5745: --- Commit 234ddb0fe1a36ad947c340114058d82c777d791f in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=234ddb0 ] NIFI-5745: When determining if backpressure should be applied across nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue This closes #3108. Signed-off-by: Koji Kawamura > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664751#comment-16664751 ] ASF GitHub Bot commented on NIFI-5745: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3108 > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
