Paul Kelly created NIFI-16098:
---------------------------------
Summary: Content can be lost due to content claim truncation
Key: NIFI-16098
URL: https://issues.apache.org/jira/browse/NIFI-16098
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.10.0
Reporter: Paul Kelly
Attachments: Content_Loss_Sample-1.json, flowscreenshot-1.png
We recently upgraded some of our servers to NiFi 2.10.0 and started seeing
MissingFlowFileException and ContentNotFoundException occasionally in some of
our flows. Reviewing the app logs, it appears to be related to NIFI-15570
because we see the content claim being truncated while the FlowFile is still
active in a different part of the flow.
Attached is a sample flow definition that reproduces the problem. Generating
two FlowFiles of different sizes (one under the
{{nifi.content.claim.max.appendable.size}} and one big enough to trigger claim
truncation), forking a copy of the FlowFiles both to a leg containing an
ExecuteGroovyScript that simply moves the FlowFile to the REL_SUCCESS
relationship which is set to terminate and to a leg that delays the FlowFile
for more than one minute causes the content in the live leg to get truncated
and lost.
!flowscreenshot.png!
This test was conducted on a brand new deployment of NiFi 2.10.0, where the
only change made to nifi.properties was to disable the content repository
archive ({{{}nifi.content.repository.archive.enabled=false{}}}) in order to
trigger truncation to occur earlier.
App logs show that the content is truncated and could no longer be found.
{code:java}
2026-07-10 00:32:35,854 INFO [NiFi Web Server-114]
o.a.n.c.s.StandardProcessScheduler Starting
RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7]
2026-07-10 00:32:35,855 INFO [NiFi Web Server-114]
o.a.n.controller.StandardProcessorNode Desired State for
RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7] now set to RUNNING
2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
o.a.n.c.s.StandardProcessScheduler Starting
GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68]
2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
o.a.n.controller.StandardProcessorNode Desired State for
GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68] now set to RUNNING
2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
o.a.n.c.s.StandardProcessScheduler Starting
CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa]
2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
o.a.n.controller.StandardProcessorNode Desired State for
CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] now set to
RUNNING
2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
o.a.n.c.s.StandardProcessScheduler Starting
ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0]
2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
o.a.n.controller.StandardProcessorNode Desired State for
ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0] now set to RUNNING
2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
o.a.n.c.s.StandardProcessScheduler Starting
GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627]
2026-07-10 00:32:35,857 INFO [Timer-Driven Process Thread-4]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7] to run with 1 threads
2026-07-10 00:32:35,857 INFO [Timer-Driven Process Thread-8]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] to run with 1
threads
2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
o.a.n.controller.StandardProcessorNode Desired State for
GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627] now set to RUNNING
2026-07-10 00:32:35,858 INFO [Timer-Driven Process Thread-10]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0] to run with 1
threads
2026-07-10 00:32:35,862 INFO [Timer-Driven Process Thread-5]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627] to run with 1 threads
2026-07-10 00:32:35,904 INFO [Timer-Driven Process Thread-3]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68] to run with 1 threads
2026-07-10 00:32:36,298 INFO [Flow Service Tasks Thread-1]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@74bb4bb5 // Another save pending =
false
2026-07-10 00:32:47,386 INFO [Checkpoint FlowFile Repository]
o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 46
Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 19
milliseconds), max Transaction ID 184
2026-07-10 00:32:47,386 INFO [Checkpoint FlowFile Repository]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 46 records in 28 milliseconds
2026-07-10 00:33:07,417 INFO [Checkpoint FlowFile Repository]
o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 126
Records and 0 Swap Files in 30 milliseconds (Stop-the-world time = 22
milliseconds), max Transaction ID 504
2026-07-10 00:33:07,417 INFO [Checkpoint FlowFile Repository]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 126 records in 30 milliseconds
2026-07-10 00:33:23,751 INFO [Write-Ahead Local State Provider Maintenance]
o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 2
Records and 0 Swap Files in 17 milliseconds (Stop-the-world time = 13
milliseconds), max Transaction ID 4
2026-07-10 00:33:25,165 INFO [FileSystemRepository Workers Thread-4]
o.a.n.c.repository.FileSystemRepository Truncating
StandardResourceClaim[id=1783657966517-22, container=default, section=22] to
20480 bytes because the FlowFile occupying the last 3145728 bytes has been
removed
...
2026-07-10 00:44:01,738 WARN [Timer-Driven Process Thread-7]
o.a.n.c.r.StandardProcessSession Unable to find content for
StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
section=22], offset=20480,
length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
dropping FlowFile
org.apache.nifi.controller.repository.ContentNotFoundException: Could not find
content for StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
section=22], offset=20480, length=3145728]: Stream contained only 0 bytes but
should have contained 3145728
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:99)
at
org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
at
org.apache.nifi.processors.standard.hash.HashService.traditionalHashStreaming(HashService.java:201)
at
org.apache.nifi.processors.standard.hash.HashService.hashValueStreaming(HashService.java:113)
at
org.apache.nifi.processors.standard.CryptographicHashContent.lambda$onTrigger$0(CryptographicHashContent.java:132)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2738)
at
org.apache.nifi.controller.repository.WeakHashMapProcessSessionFactory$FactoryRetainingProcessSession.read(WeakHashMapProcessSessionFactory.java:260)
at
org.apache.nifi.processors.standard.CryptographicHashContent.onTrigger(CryptographicHashContent.java:132)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1292)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-10 00:44:01,739 ERROR [Timer-Driven Process Thread-7]
o.a.n.p.s.CryptographicHashContent
CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] Processing
halted: yielding [1 sec]
org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
content for
StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
section=22], offset=20480,
length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
dropping FlowFile
at
org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3756)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2753)
at
org.apache.nifi.controller.repository.WeakHashMapProcessSessionFactory$FactoryRetainingProcessSession.read(WeakHashMapProcessSessionFactory.java:260)
at
org.apache.nifi.processors.standard.CryptographicHashContent.onTrigger(CryptographicHashContent.java:132)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1292)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.nifi.controller.repository.ContentNotFoundException:
Could not find content for StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
section=22], offset=20480, length=3145728]: Stream contained only 0 bytes but
should have contained 3145728
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:99)
at
org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
at
org.apache.nifi.processors.standard.hash.HashService.traditionalHashStreaming(HashService.java:201)
at
org.apache.nifi.processors.standard.hash.HashService.hashValueStreaming(HashService.java:113)
at
org.apache.nifi.processors.standard.CryptographicHashContent.lambda$onTrigger$0(CryptographicHashContent.java:132)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2738)
... 13 common frames omitted
2026-07-10 00:44:01,739 WARN [Timer-Driven Process Thread-7]
o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in
Component [CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa]]
org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
content for
StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
section=22], offset=20480,
length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
dropping FlowFile{code}
Interestingly if you replace the nearly empty ExecuteGroovyScript processor
with an empty UpdateAttribute processor that terminates the success
relationship, the problem does not occur. I have not done much testing to see
which processors can cause it, or examined what might be different about these
two processors that could cause the problem.
Disabling content claim truncation by setting
{{nifi.content.claim.truncation.enabled=false}} in nifi.properties prevents
this problem from occurring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
