Alaksiej Ščarbaty created NIFI-16136:
----------------------------------------
Summary: @OnStopped can be permanently skipped during offload
Key: NIFI-16136
URL: https://issues.apache.org/jira/browse/NIFI-16136
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 2.10.0
Reporter: Alaksiej Ščarbaty
_StandardFlowService#offload()_ stops all processors and terminates any that
report {_}getScheduledState() == ScheduledState.STOPPED{_}, without awaiting
the _Future_ returned by {_}stopProcessing(){_}:
{code:java}
flowManager.getRootGroup().stopProcessing(); // Future not awaited
flowManager.getRootGroup().findAllProcessors().stream()
.filter(pn -> pn.getScheduledState() == ScheduledState.STOPPED)
.forEach(pn -> pn.getProcessGroup().terminateProcessor(pn)); {code}
_getScheduledState()_ maps the transient _STOPPING_ state to _STOPPED_ for
backward compatibility, so _terminateProcessor()_ can run before a Processor's
stop sequence ({_}StandardProcessorNode.stop(){_}'s background runnable) has
even reached the stop hook.
_LifecycleState#terminate()_ permanently pins _activeThreadCount_ to 0.
_@OnStopped_ isn't delayed by this race, it's skipped permanently: the stop
_Future_ completes and the framework considers the Processor fully stopped, but
any cleanup in _@OnStopped_ (closing connections, releasing resources, etc.)
never runs.
*Reproduction:*
[testOffloadTerminateRaceSkipsOnStoppedPermanently in
TestStandardProcessScheduler
|https://github.com/awelless/nifi/commit/79c7ddc7a9418f1f6d643b80db6c26a68f9d5215#diff-8d6b01098138af203490bfcd481def8c309e064ec92c1d236df5b8eca3580f70R34-R833]reproduces
{_}offload(){_}'s exact sequence (block the stop thread before _unschedule()_
to ensure the race happens, call _stopProcessor()_ without awaiting its
{_}Future{_}, confirm _getScheduledState() == STOPPED_ while still physically
{_}STOPPING{_}, call {_}terminateProcessor(){_}, assert _@OnStopped_ fires). It
currently fails.
*Open questions:*
* Is it expected @OnStopped might not run when offloading happens?
* Is it expected that we don't wait for @OnStopped during offload?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
