[jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
[ https://issues.apache.org/jira/browse/NIFI-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079472#comment-17079472 ] ASF subversion and git services commented on NIFI-6849: --- Commit 683b1d995200384dcde8ec269163bc57f25b23ce in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=683b1d9 ] NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited. - Refactored Flow Synchronization to make code cleaner - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty. - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start - Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state. - Added additional log statements to aid in debugging NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior NIFI-6849: Addressed review feedback NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup This closes #3891 > On startup, NiFi should be more liberal about what it's willing to inherit > from cluster > --- > > Key: NIFI-6849 > URL: https://issues.apache.org/jira/browse/NIFI-6849 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > On startup, if an instance is configured to be clustered, the instance must > connect to the Cluster Coordinator and download the existing cluster flow, > access policies, and users and groups. The instance then performs some checks > to ensure that the local flow matches the cluster's flow. If it doesn't, then > the node fails to startup and logs errors that the local flow is different > than the cluster's flow. > This was done in order to facilitate debugging. If a particular node is not > behaving as expected, a user is able to disconnect the node from the cluster > and make modifications to the node. If the node is then restarted, it may not > be desirable to lose those changes. > However, in the vast majority of cases (probably over 98% of the time), what > the user really wants is for the node to just join back to the cluster and > inherit the cluster's flow - especially if the node just disconnected because > it failed to make a modification. This is also problematic with how the > Users, Groups, and Policies are inherited. > As a result, we should make the following modifications: > 1) If Users, Groups, or Access Policies cannot be inherited, continue to > fail, unless the flow is empty. If the flow is empty, it doesn't really make > sense to retain the authorizations' configuration because they don't really > apply to anything. As a result, if the flow is empty, just inherit whatever > the cluster has. But first, make a backup of the existing policies, users, > and groups, so that users can manually revert if they do end up needing to. > 2) If the flow differs from the cluster flow, check the proposed flow to see > if it removes any existing connections. If it does remove a connection, and > that connection has data queued locally, continue to fail. Otherwise, create > a backup of the flow and replace the node's flow with the cluster flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
[ https://issues.apache.org/jira/browse/NIFI-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079469#comment-17079469 ] ASF subversion and git services commented on NIFI-6849: --- Commit 683b1d995200384dcde8ec269163bc57f25b23ce in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=683b1d9 ] NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited. - Refactored Flow Synchronization to make code cleaner - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty. - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start - Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state. - Added additional log statements to aid in debugging NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior NIFI-6849: Addressed review feedback NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup This closes #3891 > On startup, NiFi should be more liberal about what it's willing to inherit > from cluster > --- > > Key: NIFI-6849 > URL: https://issues.apache.org/jira/browse/NIFI-6849 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > On startup, if an instance is configured to be clustered, the instance must > connect to the Cluster Coordinator and download the existing cluster flow, > access policies, and users and groups. The instance then performs some checks > to ensure that the local flow matches the cluster's flow. If it doesn't, then > the node fails to startup and logs errors that the local flow is different > than the cluster's flow. > This was done in order to facilitate debugging. If a particular node is not > behaving as expected, a user is able to disconnect the node from the cluster > and make modifications to the node. If the node is then restarted, it may not > be desirable to lose those changes. > However, in the vast majority of cases (probably over 98% of the time), what > the user really wants is for the node to just join back to the cluster and > inherit the cluster's flow - especially if the node just disconnected because > it failed to make a modification. This is also problematic with how the > Users, Groups, and Policies are inherited. > As a result, we should make the following modifications: > 1) If Users, Groups, or Access Policies cannot be inherited, continue to > fail, unless the flow is empty. If the flow is empty, it doesn't really make > sense to retain the authorizations' configuration because they don't really > apply to anything. As a result, if the flow is empty, just inherit whatever > the cluster has. But first, make a backup of the existing policies, users, > and groups, so that users can manually revert if they do end up needing to. > 2) If the flow differs from the cluster flow, check the proposed flow to see > if it removes any existing connections. If it does remove a connection, and > that connection has data queued locally, continue to fail. Otherwise, create > a backup of the flow and replace the node's flow with the cluster flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
[ https://issues.apache.org/jira/browse/NIFI-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079471#comment-17079471 ] ASF subversion and git services commented on NIFI-6849: --- Commit 683b1d995200384dcde8ec269163bc57f25b23ce in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=683b1d9 ] NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited. - Refactored Flow Synchronization to make code cleaner - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty. - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start - Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state. - Added additional log statements to aid in debugging NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior NIFI-6849: Addressed review feedback NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup This closes #3891 > On startup, NiFi should be more liberal about what it's willing to inherit > from cluster > --- > > Key: NIFI-6849 > URL: https://issues.apache.org/jira/browse/NIFI-6849 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > On startup, if an instance is configured to be clustered, the instance must > connect to the Cluster Coordinator and download the existing cluster flow, > access policies, and users and groups. The instance then performs some checks > to ensure that the local flow matches the cluster's flow. If it doesn't, then > the node fails to startup and logs errors that the local flow is different > than the cluster's flow. > This was done in order to facilitate debugging. If a particular node is not > behaving as expected, a user is able to disconnect the node from the cluster > and make modifications to the node. If the node is then restarted, it may not > be desirable to lose those changes. > However, in the vast majority of cases (probably over 98% of the time), what > the user really wants is for the node to just join back to the cluster and > inherit the cluster's flow - especially if the node just disconnected because > it failed to make a modification. This is also problematic with how the > Users, Groups, and Policies are inherited. > As a result, we should make the following modifications: > 1) If Users, Groups, or Access Policies cannot be inherited, continue to > fail, unless the flow is empty. If the flow is empty, it doesn't really make > sense to retain the authorizations' configuration because they don't really > apply to anything. As a result, if the flow is empty, just inherit whatever > the cluster has. But first, make a backup of the existing policies, users, > and groups, so that users can manually revert if they do end up needing to. > 2) If the flow differs from the cluster flow, check the proposed flow to see > if it removes any existing connections. If it does remove a connection, and > that connection has data queued locally, continue to fail. Otherwise, create > a backup of the flow and replace the node's flow with the cluster flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
[ https://issues.apache.org/jira/browse/NIFI-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079470#comment-17079470 ] ASF subversion and git services commented on NIFI-6849: --- Commit 683b1d995200384dcde8ec269163bc57f25b23ce in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=683b1d9 ] NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited. - Refactored Flow Synchronization to make code cleaner - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty. - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start - Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state. - Added additional log statements to aid in debugging NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior NIFI-6849: Addressed review feedback NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup This closes #3891 > On startup, NiFi should be more liberal about what it's willing to inherit > from cluster > --- > > Key: NIFI-6849 > URL: https://issues.apache.org/jira/browse/NIFI-6849 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > On startup, if an instance is configured to be clustered, the instance must > connect to the Cluster Coordinator and download the existing cluster flow, > access policies, and users and groups. The instance then performs some checks > to ensure that the local flow matches the cluster's flow. If it doesn't, then > the node fails to startup and logs errors that the local flow is different > than the cluster's flow. > This was done in order to facilitate debugging. If a particular node is not > behaving as expected, a user is able to disconnect the node from the cluster > and make modifications to the node. If the node is then restarted, it may not > be desirable to lose those changes. > However, in the vast majority of cases (probably over 98% of the time), what > the user really wants is for the node to just join back to the cluster and > inherit the cluster's flow - especially if the node just disconnected because > it failed to make a modification. This is also problematic with how the > Users, Groups, and Policies are inherited. > As a result, we should make the following modifications: > 1) If Users, Groups, or Access Policies cannot be inherited, continue to > fail, unless the flow is empty. If the flow is empty, it doesn't really make > sense to retain the authorizations' configuration because they don't really > apply to anything. As a result, if the flow is empty, just inherit whatever > the cluster has. But first, make a backup of the existing policies, users, > and groups, so that users can manually revert if they do end up needing to. > 2) If the flow differs from the cluster flow, check the proposed flow to see > if it removes any existing connections. If it does remove a connection, and > that connection has data queued locally, continue to fail. Otherwise, create > a backup of the flow and replace the node's flow with the cluster flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
[ https://issues.apache.org/jira/browse/NIFI-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012253#comment-17012253 ] Joe Witt commented on NIFI-6849: On further review I punted it from the 1.11 RC as this is too big/too risky and needs a good thorough review. Certainly a good candidate for a 1.12 release. > On startup, NiFi should be more liberal about what it's willing to inherit > from cluster > --- > > Key: NIFI-6849 > URL: https://issues.apache.org/jira/browse/NIFI-6849 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > On startup, if an instance is configured to be clustered, the instance must > connect to the Cluster Coordinator and download the existing cluster flow, > access policies, and users and groups. The instance then performs some checks > to ensure that the local flow matches the cluster's flow. If it doesn't, then > the node fails to startup and logs errors that the local flow is different > than the cluster's flow. > This was done in order to facilitate debugging. If a particular node is not > behaving as expected, a user is able to disconnect the node from the cluster > and make modifications to the node. If the node is then restarted, it may not > be desirable to lose those changes. > However, in the vast majority of cases (probably over 98% of the time), what > the user really wants is for the node to just join back to the cluster and > inherit the cluster's flow - especially if the node just disconnected because > it failed to make a modification. This is also problematic with how the > Users, Groups, and Policies are inherited. > As a result, we should make the following modifications: > 1) If Users, Groups, or Access Policies cannot be inherited, continue to > fail, unless the flow is empty. If the flow is empty, it doesn't really make > sense to retain the authorizations' configuration because they don't really > apply to anything. As a result, if the flow is empty, just inherit whatever > the cluster has. But first, make a backup of the existing policies, users, > and groups, so that users can manually revert if they do end up needing to. > 2) If the flow differs from the cluster flow, check the proposed flow to see > if it removes any existing connections. If it does remove a connection, and > that connection has data queued locally, continue to fail. Otherwise, create > a backup of the flow and replace the node's flow with the cluster flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
[ https://issues.apache.org/jira/browse/NIFI-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009038#comment-17009038 ] Joe Witt commented on NIFI-6849: This does not have a reviewer and is a big change. But the benefit to typical/desired usage patterns is huge. I'm pulling this into 1.11 for now and will hunt down a reviewer. > On startup, NiFi should be more liberal about what it's willing to inherit > from cluster > --- > > Key: NIFI-6849 > URL: https://issues.apache.org/jira/browse/NIFI-6849 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > On startup, if an instance is configured to be clustered, the instance must > connect to the Cluster Coordinator and download the existing cluster flow, > access policies, and users and groups. The instance then performs some checks > to ensure that the local flow matches the cluster's flow. If it doesn't, then > the node fails to startup and logs errors that the local flow is different > than the cluster's flow. > This was done in order to facilitate debugging. If a particular node is not > behaving as expected, a user is able to disconnect the node from the cluster > and make modifications to the node. If the node is then restarted, it may not > be desirable to lose those changes. > However, in the vast majority of cases (probably over 98% of the time), what > the user really wants is for the node to just join back to the cluster and > inherit the cluster's flow - especially if the node just disconnected because > it failed to make a modification. This is also problematic with how the > Users, Groups, and Policies are inherited. > As a result, we should make the following modifications: > 1) If Users, Groups, or Access Policies cannot be inherited, continue to > fail, unless the flow is empty. If the flow is empty, it doesn't really make > sense to retain the authorizations' configuration because they don't really > apply to anything. As a result, if the flow is empty, just inherit whatever > the cluster has. But first, make a backup of the existing policies, users, > and groups, so that users can manually revert if they do end up needing to. > 2) If the flow differs from the cluster flow, check the proposed flow to see > if it removes any existing connections. If it does remove a connection, and > that connection has data queued locally, continue to fail. Otherwise, create > a backup of the flow and replace the node's flow with the cluster flow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
