Xinyu Wang created NIFI-16584:
---------------------------------
Summary: One blocked property validator (e.g.
DirectoryExistsValidator on an unresponsive NFS mount) stalls the periodic
validation pass indefinitely
Key: NIFI-16584
URL: https://issues.apache.org/jira/browse/NIFI-16584
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 2.5.0
Reporter: Xinyu Wang
{code}
h3. Summary
The periodic validation pass ({{TriggerValidationTask}}) runs all component
validations serially on a single "Validate Components" thread, and each
validation calls property validators synchronously with no timeout. If one
validator performs filesystem I/O on a path that hangs (for example, a
directory on a {{hard}}-mounted NFS share whose server session is broken), that
thread blocks forever and the pass never completes. Because the task is
scheduled with {{scheduleWithFixedDelay}}, no further pass ever starts.
h3. Observed behavior
On a 2.5.0 node, a thread dump showed:
{noformat}
"Validate Components Thread-1" RUNNABLE
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.File.exists(File.java:836)
at
org.apache.nifi.processor.util.StandardValidators$DirectoryExistsValidator.validate(StandardValidators.java:953)
at
org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:180)
at
org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:916)
at
org.apache.nifi.controller.StandardProcessorNode.computeValidationErrors(StandardProcessorNode.java:1058)
at
org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:851)
at
org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52)
at
org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42)
{noformat}
At the kernel level the thread was in uninterruptible sleep ({{D}} state) with
wchan {{rpc_wait_bit_killable}}, and its CPU time did not change over 18
minutes. The NFSv4.1 lease for the affected mount had expired about 45 hours
earlier ({{lease_expired}} in {{/proc/<pid>/mountstats}}). The same mount also
had 8 {{Timer-Driven Process Thread}} threads stuck in {{stat}}/{{access}}
calls. Nothing is logged, because a {{hard}} mount never returns an error.
The other four threads of the validation pool were idle (parked on the
executor's {{DelayedWorkQueue}}).
h3. Why this is a problem
- {{FlowController}} creates a 5-thread pool ({{FlowEngine(5, "Validate
Components", true)}}) but schedules only one {{TriggerValidationTask}} on it
({{scheduleWithFixedDelay(..., 5, 5, SECONDS)}}).
- {{TriggerValidationTask.run()}} iterates controller services, reporting
tasks, flow analysis rules, parameter providers, all processors, registry
clients and connectors sequentially, calling
{{validationTrigger.trigger(node)}}, which runs {{performValidation()}}
synchronously on the calling thread.
- A single hung validator therefore blocks the whole periodic validation pass.
Components whose validity depends on the environment (the case NIFI-98
describes for {{FileExistsValidator}}) are never re-validated again, and
nothing is logged or surfaced to the user.
- Components that are newly created or modified go through
{{resetValidationState()}} and {{triggerAsync()}} and can still be validated by
the idle pool threads, so the impact is limited to the periodic validation
pass, but it is silent and permanent until restart.
h3. Expected behavior
A hung or very slow validation of one component should not prevent the
validation of other components, and should be visible to operators.
h3. Possible approaches
Submit each component's validation to the validation thread pool (as
{{triggerAsync}} already does) instead of running it inline in the periodic
validation pass, so one blocked component occupies only one pool thread.
Add a per-component timeout for validation and report the component as INVALID
(with an explanation such as "validation timed out") so the pass can continue.
NIFI-15121 took a similar approach for {{ExecuteGroovyScript}} using a virtual
thread and a timeout.
Skip components whose previous validation is still in progress, so the pass
does not pile up on the same component.
Log a warning when a validation exceeds a threshold, so a hung validator is
diagnosable without a thread dump.
h3. Steps to reproduce (not yet verified by the reporter)
Mount an NFS export with {{hard}} on the NiFi host.
Add a {{ListFile}} (or any processor using {{DirectoryExistsValidator}}) with
{{Input Directory}} pointing at that mount.
Make the NFS server unreachable, for example by dropping its traffic with a
firewall rule.
Wait for the next periodic validation pass to reach that processor, then take a
thread dump and check the "Validate Components" threads.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)