[
https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205013#comment-15205013
]
ASF GitHub Bot commented on NIFI-1563:
--------------------------------------
GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/294
NIFI-1563: Federate requests and merge responses from nodes instead o…
…f storing bulletins and stats at NCM
- Updating UI to support restructured status history DTO.
Return 'Insufficient History' message if aggregate stats don't have enough
data points, even if all nodes do (which can be the case if the node performing
the aggregation has a different value for the
'nifi.components.status.snapshot.frequency' property than the other nodes)
Bug fixes; code cleanup; replicate requests to bulletin board endpoint
Refactored the <Component>StatusDTO objects into <Component>StatusDTO,
<Component>StatusSnapshotDTO, Node<Component>StatusSnapshotDTO objects
- Introducing endpoints for accessing individual component status.
- Wiring up new endpoints and updated core.
- Code clean up.
- Starting to handling status merging of individual components.
- Nodewise breakdown has been added to Processors but the remaining
components still need to be updated.
Refactor so that System Diagnostics requests are replicated to nodes
instead of the information being pulled from Heartbeats
Replicate request for counters instead of pulling them from heartbeats
Removed the getCounters / setCounters method from HeartbeatPayload
- Implementing component specific endpoints.
- Removing unused endpoints.
- Supporting nodewise breakdown for system diagnostics and counters.
- Updating DTOs to use more consistent naming.
- Code clean up.
- Addressing contrib issues.
Removed ProcessGroupStatus from HeartbeatPayload
- Removing nodewise from the system diagnostics endpoint. Had included it
for testing that option but did not intend for it to be committed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi NIFI-1563
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #294
----
commit 1bbe0aa7f7c80d13a67178261b88551389a92f17
Author: Mark Payne <marka...@hotmail.com>
Date: 2016-03-03T18:29:34Z
NIFI-1563: Federate requests and merge responses from nodes instead of
storing bulletins and stats at NCM
- Updating UI to support restructured status history DTO.
Return 'Insufficient History' message if aggregate stats don't have enough
data points, even if all nodes do (which can be the case if the node performing
the aggregation has a different value for the
'nifi.components.status.snapshot.frequency' property than the other nodes)
Bug fixes; code cleanup; replicate requests to bulletin board endpoint
Refactored the <Component>StatusDTO objects into <Component>StatusDTO,
<Component>StatusSnapshotDTO, Node<Component>StatusSnapshotDTO objects
- Introducing endpoints for accessing individual component status.
- Wiring up new endpoints and updated core.
- Code clean up.
- Starting to handling status merging of individual components.
- Nodewise breakdown has been added to Processors but the remaining
components still need to be updated.
Refactor so that System Diagnostics requests are replicated to nodes
instead of the information being pulled from Heartbeats
Replicate request for counters instead of pulling them from heartbeats
Removed the getCounters / setCounters method from HeartbeatPayload
- Implementing component specific endpoints.
- Removing unused endpoints.
- Supporting nodewise breakdown for system diagnostics and counters.
- Updating DTOs to use more consistent naming.
- Code clean up.
- Addressing contrib issues.
Removed ProcessGroupStatus from HeartbeatPayload
- Removing nodewise from the system diagnostics endpoint. Had included it
for testing that option but did not intend for it to be committed.
----
> Remove storage of components' stats and bulletins from NCM
> ----------------------------------------------------------
>
> Key: NIFI-1563
> URL: https://issues.apache.org/jira/browse/NIFI-1563
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 1.0.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the
> stats for all components in the node. This happens every 5 seconds by
> default. This results in quite a lot of chatter between the NCM and nodes. It
> made sense to take this approach when the clustering concept was designed
> because there were no Process Groups, and we had no notion of merging
> responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the
> responses on demand, rather than storing this information on the NCM. This
> requires far less bandwidth because we only need to pull the stats for a
> particular Process Group and only on demand instead of every 5 seconds.
> Additionally, this is laying groundwork for the Zero-Master clustering that
> we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the
> bulletins and stats history. These requests will all need to be federated and
> the responses merged on-demand.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
