[jira] [Commented] (NIFI-9689) Do not schedule processors when all FlowFiles are penalized
[ https://issues.apache.org/jira/browse/NIFI-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500065#comment-17500065 ] ASF subversion and git services commented on NIFI-9689: --- Commit 53a35ae4c999824b1c79c31954d2b9b719efd997 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=53a35ae ] NIFI-9689: When checking FlowFile Availability, consider swap queue and trigger data to be swapped in, since calling poll() will no longer happen if no data is available Signed-off-by: Joe Gresock This closes #5821. > Do not schedule processors when all FlowFiles are penalized > --- > > Key: NIFI-9689 > URL: https://issues.apache.org/jira/browse/NIFI-9689 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.16.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When all FlowFiles in a given queue are penalized, the destination processor > is still scheduled to run. As a result, the Processor is triggered, no data > is available, and it generally returns. However, this means that the UI shows > millions of tasks completed over a short period of time, when no work was > actually done. > Further, it means that we constantly schedule the processor to run as fast as > it can, which wastes compute resources. Instead, we should detect that the > head of the queue is penalized (and therefore all FlowFiles in the queue are > penalized), and treat it the same as we would if the queue were empty, which > allows the "bored yield duration" to take effect, significantly reducing CPU > cycles. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9689) Do not schedule processors when all FlowFiles are penalized
[ https://issues.apache.org/jira/browse/NIFI-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493495#comment-17493495 ] ASF subversion and git services commented on NIFI-9689: --- Commit 2aa6bd1e135cd4959cef5a483c454dc9a1591439 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2aa6bd1 ] NIFI-9689: When all FlowFiles in a FlowFile Queue are penalized, do not schedule the destination to run. Also expose this fact via the ConnectionStatusSnapshotDTO, as this allows the front-end to render this information to the user in order to avoid confusion when it appears that the Processor has data but does nothing Signed-off-by: Matthew Burgess This closes #5771 > Do not schedule processors when all FlowFiles are penalized > --- > > Key: NIFI-9689 > URL: https://issues.apache.org/jira/browse/NIFI-9689 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When all FlowFiles in a given queue are penalized, the destination processor > is still scheduled to run. As a result, the Processor is triggered, no data > is available, and it generally returns. However, this means that the UI shows > millions of tasks completed over a short period of time, when no work was > actually done. > Further, it means that we constantly schedule the processor to run as fast as > it can, which wastes compute resources. Instead, we should detect that the > head of the queue is penalized (and therefore all FlowFiles in the queue are > penalized), and treat it the same as we would if the queue were empty, which > allows the "bored yield duration" to take effect, significantly reducing CPU > cycles. -- This message was sent by Atlassian Jira (v8.20.1#820001)
