[
https://issues.apache.org/jira/browse/NIFI-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110920#comment-18110920
]
ASF subversion and git services commented on NIFI-16284:
--------------------------------------------------------
Commit 374e7cc68b961c0941a517b52543f4720969fffe in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=374e7cc68b9 ]
NIFI-16284 Wait for Load Balancing to complete in ClusteredConnectorBacklogIT
(#11615)
- Complete Process Group drop requests only after every connection finishes
- Describe aggregate drop request lifecycle in StandardProcessGroup
Signed-off-by: David Handermann <[email protected]>
> Flaky System Test
> ClusteredConnectorBacklogIT.testClusteredReadQueueAttributesLoadBalancedConnection
> ----------------------------------------------------------------------------------------------------
>
> Key: NIFI-16284
> URL: https://issues.apache.org/jira/browse/NIFI-16284
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h3. Problem
> {{ClusteredConnectorBacklogIT#testClusteredReadQueueAttributesLoadBalancedConnection}}
> fails intermittently in CI during test teardown, not in the test body itself:
> {code}
> org.apache.nifi.toolkit.client.NiFiClientException: Error deleting Connector:
> Node localhost:5671 is unable to fulfill this request due to: Cannot delete
> StandardConnectorNode[...name=BacklogReportingTestConnector, state=STOPPED]
> because its Process Group has 82 FlowFiles queued; all FlowFiles must be
> removed before it can be deleted.
> {code}
> Observed on CI run 33395377036 (ubuntu-24.04, Java 25), surfaced while
> reviewing PR [#11599|https://github.com/apache/nifi/pull/11599]. The sibling
> test
> {{testClusteredReadQueueAttributesNormalConnection}}
> ({{DO_NOT_LOAD_BALANCE}}) has
> not exhibited this failure - only the {{ROUND_ROBIN}} variant has, which
> points
> toward Load Balancing as a factor.
> h3. Suspected Root Cause
> This is a timing-dependent failure that we have not been able to reproduce
> on demand, so the following is our best working theory rather than a
> confirmed diagnosis.
> {{NiFiSystemIT#teardown()}} -> {{destroyFlow()}} ->
> {{NiFiClientUtil#deleteConnectors()}}
> calls {{purgeConnectorFlowFiles()}} (a drop request) immediately before
> deleting
> each Connector. For a load-balanced connection
> ({{SocketLoadBalancedFlowFileQueue}}), FlowFiles can plausibly be "in flight"
> between nodes: a background thread polls a FlowFile out of the local
> {{RemoteQueuePartition}} to transmit it to the peer node, and the record does
> not appear to be counted as present on either node's queue until the
> transfer completes and the peer enqueues it. If a drop/purge request is
> issued during that window, it could report completion without having
> dropped those in-transit records, which would then land on the peer's queue
> after the purge already ran - leaving FlowFiles queued when the connector
> delete is subsequently attempted. This lines up with the symptom but has
> not been directly observed happening.
> {{ConnectionDTO#getLoadBalanceStatus()}} ({{LOAD_BALANCE_ACTIVE}} /
> {{LOAD_BALANCE_INACTIVE}} / {{LOAD_BALANCE_NOT_CONFIGURED}}, backed by
> {{FlowFileQueue#isActivelyLoadBalancing()}}) is the existing public signal for
> this in-flight state.
> h3. Investigation notes
> Checked whether this could be related to NIFI-16262 (PR
> [#11597|https://github.com/apache/nifi/pull/11597], "Fix
> Connector stop handling for multiple callers"). That change only touches
> start/stop race handling in {{StandardConnectorNode}} and does not touch
> purge/drop-request handling, so it looks unrelated. PR
> [#11599|https://github.com/apache/nifi/pull/11599]'s base
> commit already includes the NIFI-16262 fix, and the failure still
> reproduced there, which is consistent with the two being unrelated.
> h3. Proposed Mitigation
> Attempted to stabilize the test itself (no framework changes) in
> {{ClusteredConnectorBacklogIT}}: wrapped the test body in try/finally and
> added
> a private {{waitForLoadBalancingComplete()}} helper that polls each connection
> in the Connector's flow via {{getConnectorClient().getFlow()}} until none
> report
> {{LOAD_BALANCE_ACTIVE}}. This runs in the finally block, after
> {{stopConnector()}},
> with the intent that Load Balancing has settled before control returns to
> teardown's purge - regardless of whether the test's own assertions passed.
> This is a hypothesis-driven mitigation based on the suspected root cause
> above, not a confirmed fix, since the original failure could not be
> reproduced on demand to verify against directly. Kept local to this test
> class rather than added as a general-purpose helper on {{NiFiSystemIT}}, since
> the race is specific to this test's use of a load-balanced Connector queue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
