[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357295#comment-15357295 ] Mark Payne commented on NIFI-1563: -- [~mcgilman] - I believe we can close this ticket now, as the bulletins are working now. Do you agree? > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224453#comment-15224453 ] ASF subversion and git services commented on NIFI-1563: --- Commit 0d3bd2c401aacc5bcd602afe90883d751ff773f3 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0d3bd2c ] NIFI-1563: - Federate requests and merge responses from nodes instead of storing bulletins and stats at NCM - Updating UI to support restructured status history DTO. - Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) - Bug fixes; code cleanup; replicate requests to bulletin board endpoint - Refactored the StatusDTO objects into StatusDTO, StatusSnapshotDTO, NodeStatusSnapshotDTO objects - Introducing endpoints for accessing individual component status. - Wiring up new endpoints and updated core. - Code clean up. - Starting to handling status merging of individual components. - Nodewise breakdown has been added to Processors but the remaining components still need to be updated. - Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats - Replicate request for counters instead of pulling them from heartbeats - Removed the getCounters / setCounters method from HeartbeatPayload - Implementing component specific endpoints. - Removing unused endpoints. - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. - Removed ProcessGroupStatus from HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. - Addressing comments in PR #294. - This closes #294 Signed-off-by: Matt Gilman > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208712#comment-15208712 ] ASF GitHub Bot commented on NIFI-1563: -- Github user markap14 closed the pull request at: https://github.com/apache/nifi/pull/294 > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208695#comment-15208695 ] ASF subversion and git services commented on NIFI-1563: --- Commit 137868461860132097e01fdc32e583c33570 in nifi's branch refs/heads/1.x from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=137 ] NIFI-1563: - Federate requests and merge responses from nodes instead of storing bulletins and stats at NCM - Updating UI to support restructured status history DTO. - Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) - Bug fixes; code cleanup; replicate requests to bulletin board endpoint - Refactored the StatusDTO objects into StatusDTO, StatusSnapshotDTO, NodeStatusSnapshotDTO objects - Introducing endpoints for accessing individual component status. - Wiring up new endpoints and updated core. - Code clean up. - Starting to handling status merging of individual components. - Nodewise breakdown has been added to Processors but the remaining components still need to be updated. - Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats - Replicate request for counters instead of pulling them from heartbeats - Removed the getCounters / setCounters method from HeartbeatPayload - Implementing component specific endpoints. - Removing unused endpoints. - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. - Removed ProcessGroupStatus from HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. - Addressing comments in PR #294. - This closes #294 > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208437#comment-15208437 ] ASF GitHub Bot commented on NIFI-1563: -- Github user markap14 commented on the pull request: https://github.com/apache/nifi/pull/294#issuecomment-200355764 @mcgilman I looked at the commit that you proposed. Looks good! +1. Thanks for jumping in there and fixing that! > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207204#comment-15207204 ] ASF GitHub Bot commented on NIFI-1563: -- Github user mcgilman commented on the pull request: https://github.com/apache/nifi/pull/294#issuecomment-23951 @markap14 Proposed commit for my comments [1]. If this looks good to you, I'll include them in your PR. [1] https://github.com/mcgilman/nifi/commit/9796e7620cb064653293d8bc0b2293b8b063f3b7 > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206436#comment-15206436 ] ASF GitHub Bot commented on NIFI-1563: -- Github user mcgilman commented on the pull request: https://github.com/apache/nifi/pull/294#issuecomment-199837163 Also, in nf-cluster-search.js we can delete line 72 and 172. We no longer need to set: `nf.SummaryTable.systemDiagnosticsUrl` Setting the clusterNodeId on the preceding line should be sufficient. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206428#comment-15206428 ] ASF GitHub Bot commented on NIFI-1563: -- Github user mcgilman commented on the pull request: https://github.com/apache/nifi/pull/294#issuecomment-199835774 @markap14 Unfortunately, the changeset is too big for me to comment on directly. But in nf-counters-table.js the Ajax request on line 281 does not need to set the nodewise flag to true. We do not currently provide a node by node breakdown of the Counters. When we do offer that, we should add that flag back in. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[
https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206392#comment-15206392
]
ASF GitHub Bot commented on NIFI-1563:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/294#discussion_r56988394
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/CounterDTO.java
---
@@ -98,4 +98,12 @@ public void setValueCount(Long valueCount) {
this.valueCount = valueCount;
}
+@Override
+public CounterDTO clone() {
--- End diff --
@markap14 There appears to be about 5 instances of super.clone(). Given the
ambiguous specification on the implementation of Object.clone and the
inconsistency of doing a manual clone explicitly elsewhere can we update these?
> Remove storage of components' stats and bulletins from NCM
> --
>
> Key: NIFI-1563
> URL: https://issues.apache.org/jira/browse/NIFI-1563
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.0.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the
> stats for all components in the node. This happens every 5 seconds by
> default. This results in quite a lot of chatter between the NCM and nodes. It
> made sense to take this approach when the clustering concept was designed
> because there were no Process Groups, and we had no notion of merging
> responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the
> responses on demand, rather than storing this information on the NCM. This
> requires far less bandwidth because we only need to pull the stats for a
> particular Process Group and only on demand instead of every 5 seconds.
> Additionally, this is laying groundwork for the Zero-Master clustering that
> we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the
> bulletins and stats history. These requests will all need to be federated and
> the responses merged on-demand.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[
https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206336#comment-15206336
]
ASF GitHub Bot commented on NIFI-1563:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/294#discussion_r56984358
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/CounterDTO.java
---
@@ -98,4 +98,12 @@ public void setValueCount(Long valueCount) {
this.valueCount = valueCount;
}
+@Override
+public CounterDTO clone() {
--- End diff --
Was this the intended implementation of this method? This implementation
doesn't seem consistent with others.
> Remove storage of components' stats and bulletins from NCM
> --
>
> Key: NIFI-1563
> URL: https://issues.apache.org/jira/browse/NIFI-1563
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.0.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the
> stats for all components in the node. This happens every 5 seconds by
> default. This results in quite a lot of chatter between the NCM and nodes. It
> made sense to take this approach when the clustering concept was designed
> because there were no Process Groups, and we had no notion of merging
> responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the
> responses on demand, rather than storing this information on the NCM. This
> requires far less bandwidth because we only need to pull the stats for a
> particular Process Group and only on demand instead of every 5 seconds.
> Additionally, this is laying groundwork for the Zero-Master clustering that
> we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the
> bulletins and stats history. These requests will all need to be federated and
> the responses merged on-demand.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205036#comment-15205036 ] ASF GitHub Bot commented on NIFI-1563: -- Github user mcgilman commented on a diff in the pull request: https://github.com/apache/nifi/pull/294#discussion_r56890522 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/pom.xml --- @@ -26,5 +26,13 @@ com.wordnik swagger-annotations + +org.apache.nifi +nifi-utils + + +org.apache.nifi +nifi-api + --- End diff -- Are these dependencies necessary? Would like to have this artifact as 'light' and 'portability' as possible. Should just be definitions of transport objects for using the REST API. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205018#comment-15205018 ] ASF GitHub Bot commented on NIFI-1563: -- Github user mcgilman commented on the pull request: https://github.com/apache/nifi/pull/294#issuecomment-199458161 Reviewing... > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205013#comment-15205013 ] ASF GitHub Bot commented on NIFI-1563: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/294 NIFI-1563: Federate requests and merge responses from nodes instead o… …f storing bulletins and stats at NCM - Updating UI to support restructured status history DTO. Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) Bug fixes; code cleanup; replicate requests to bulletin board endpoint Refactored the StatusDTO objects into StatusDTO, StatusSnapshotDTO, NodeStatusSnapshotDTO objects - Introducing endpoints for accessing individual component status. - Wiring up new endpoints and updated core. - Code clean up. - Starting to handling status merging of individual components. - Nodewise breakdown has been added to Processors but the remaining components still need to be updated. Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats Replicate request for counters instead of pulling them from heartbeats Removed the getCounters / setCounters method from HeartbeatPayload - Implementing component specific endpoints. - Removing unused endpoints. - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. Removed ProcessGroupStatus from HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-1563 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #294 commit 1bbe0aa7f7c80d13a67178261b88551389a92f17 Author: Mark Payne Date: 2016-03-03T18:29:34Z NIFI-1563: Federate requests and merge responses from nodes instead of storing bulletins and stats at NCM - Updating UI to support restructured status history DTO. Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) Bug fixes; code cleanup; replicate requests to bulletin board endpoint Refactored the StatusDTO objects into StatusDTO, StatusSnapshotDTO, NodeStatusSnapshotDTO objects - Introducing endpoints for accessing individual component status. - Wiring up new endpoints and updated core. - Code clean up. - Starting to handling status merging of individual components. - Nodewise breakdown has been added to Processors but the remaining components still need to be updated. Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats Replicate request for counters instead of pulling them from heartbeats Removed the getCounters / setCounters method from HeartbeatPayload - Implementing component specific endpoints. - Removing unused endpoints. - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. Removed ProcessGroupStatus from HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. Th
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204412#comment-15204412 ] ASF subversion and git services commented on NIFI-1563: --- Commit d85d9f768af39491b732654565db3b13cb167a62 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d85d9f7 ] NIFI-1563: - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204246#comment-15204246 ] ASF subversion and git services commented on NIFI-1563: --- Commit ec4b1cebc90e5c70ae3962cfae9db8b4ba6f17c7 in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ec4b1ce ] NIFI-1563: Removed ProcessGroupStatus from HeartbeatPayload > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199824#comment-15199824 ] ASF subversion and git services commented on NIFI-1563: --- Commit e49636af5e3b9be830ca6f6c8a5148d20620a06d in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e49636a ] NIFI-1563: Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200172#comment-15200172 ] ASF subversion and git services commented on NIFI-1563: --- Commit a531a3b3daf7b470876c379874b3b7aae1e93ef4 in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a531a3b ] NIFI-1563: Removed the getCounters / setCounters method from HeartbeatPayload > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200130#comment-15200130 ] ASF subversion and git services commented on NIFI-1563: --- Commit c335d810fb3dd05c80692caafede0a29621af357 in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c335d81 ] NIFI-1563: Replicate request for counters instead of pulling them from heartbeats > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201620#comment-15201620 ] ASF subversion and git services commented on NIFI-1563: --- Commit 61a7cbeaa1e049b24998352689996710433c0629 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=61a7cbe ] NIFI-1563: - Implementing component specific endpoints. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202095#comment-15202095 ] ASF subversion and git services commented on NIFI-1563: --- Commit 8f28237ff32acda4e937c8a06417019554657692 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8f28237 ] NIFI-1563: - Removing unused endpoints. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202096#comment-15202096 ] ASF subversion and git services commented on NIFI-1563: --- Commit 3337ea9513e32528838617c3649b9a00e9f4817a in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3337ea9 ] NIFI-1563: - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190028#comment-15190028 ] ASF subversion and git services commented on NIFI-1563: --- Commit 19e8d7db93f0e3fd3fbcb83e62c564ffe95fba11 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=19e8d7d ] NIFI-1563: - Wiring up new endpoints and updated core. - Code clean up. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187885#comment-15187885 ] ASF subversion and git services commented on NIFI-1563: --- Commit 7a6dc4a7ec36025ba4719ec0990ee0d514630d44 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7a6dc4a ] NIFI-1563: - Introducing endpoints for accessing individual component status. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187614#comment-15187614 ] ASF subversion and git services commented on NIFI-1563: --- Commit deba41d8e0ae86208f0c1d5a50a191f14c1b9ab1 in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=deba41d ] NIFI-1563: Bug fixes; code cleanup; replicate requests to bulletin board endpoint > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187615#comment-15187615 ] ASF subversion and git services commented on NIFI-1563: --- Commit a901bc65fc1c4bed6a1dc7b9aa097ff856734022 in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a901bc6 ] NIFI-1563: Refactored the StatusDTO objects into StatusDTO, StatusSnapshotDTO, NodeStatusSnapshotDTO objects > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183128#comment-15183128 ] ASF subversion and git services commented on NIFI-1563: --- Commit 7f49f8723851792a19304304351a08eca5e61a8d in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7f49f87 ] NIFI-1563: Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180589#comment-15180589 ] ASF subversion and git services commented on NIFI-1563: --- Commit 8f187274afdbfa08538c3b867aad664716bc8a44 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8f18727 ] NIFI-1563: - Updating UI to support restructured status history DTO. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178317#comment-15178317 ] ASF subversion and git services commented on NIFI-1563: --- Commit ad32cb8243cc20ce7b04cc1fce0263503322ce5b in nifi's branch refs/heads/NIFI-1563 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ad32cb8 ] NIFI-1563: Federate requests and merge responses from nodes instead of storing bulletins and stats at NCM > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167253#comment-15167253 ] Mark Payne commented on NIFI-1563: -- Will also require requests for Counters & System Diagnostics to be federated. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
