[
https://issues.apache.org/jira/browse/NIFI-16098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-16098:
----------------------------------
Status: Patch Available (was: In Progress)
> Content can be lost due to content claim truncation
> ---------------------------------------------------
>
> Key: NIFI-16098
> URL: https://issues.apache.org/jira/browse/NIFI-16098
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.10.0
> Reporter: Paul Kelly
> Assignee: Pierre Villard
> Priority: Major
> Attachments: Content_Loss_Sample-1.json, flowscreenshot-1.png
>
>
> We recently upgraded some of our servers to NiFi 2.10.0 and started seeing
> MissingFlowFileException and ContentNotFoundException occasionally in some of
> our flows. Reviewing the app logs, it appears to be related to NIFI-15570
> because we see the content claim being truncated while the FlowFile is still
> active in a different part of the flow.
> Attached is a sample flow definition that reproduces the problem. Generating
> two FlowFiles of different sizes (one under the
> {{nifi.content.claim.max.appendable.size}} and one big enough to trigger
> claim truncation), forking a copy of the FlowFiles both to a leg containing
> an ExecuteGroovyScript that simply moves the FlowFile to the REL_SUCCESS
> relationship which is set to terminate and to a leg that delays the FlowFile
> for more than one minute causes the content in the live leg to get truncated
> and lost.
> !flowscreenshot.png!
> This test was conducted on a brand new deployment of NiFi 2.10.0, where the
> only change made to nifi.properties was to disable the content repository
> archive ({{{}nifi.content.repository.archive.enabled=false{}}}) in order to
> trigger truncation to occur earlier.
> App logs show that the content is truncated and could no longer be found.
>
> {code:java}
> 2026-07-10 00:32:35,854 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7]
> 2026-07-10 00:32:35,855 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7] now set to RUNNING
> 2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68]
> 2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68] now set to RUNNING
> 2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa]
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] now set to
> RUNNING
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0]
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0] now set to
> RUNNING
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627]
> 2026-07-10 00:32:35,857 INFO [Timer-Driven Process Thread-4]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7] to run with 1 threads
> 2026-07-10 00:32:35,857 INFO [Timer-Driven Process Thread-8]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] to run with
> 1 threads
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627] now set to RUNNING
> 2026-07-10 00:32:35,858 INFO [Timer-Driven Process Thread-10]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0] to run with 1
> threads
> 2026-07-10 00:32:35,862 INFO [Timer-Driven Process Thread-5]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627] to run with 1
> threads
> 2026-07-10 00:32:35,904 INFO [Timer-Driven Process Thread-3]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68] to run with 1
> threads
> 2026-07-10 00:32:36,298 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@74bb4bb5 // Another save pending =
> false
> 2026-07-10 00:32:47,386 INFO [Checkpoint FlowFile Repository]
> o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 46
> Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 19
> milliseconds), max Transaction ID 184
> 2026-07-10 00:32:47,386 INFO [Checkpoint FlowFile Repository]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 46 records in 28 milliseconds
> 2026-07-10 00:33:07,417 INFO [Checkpoint FlowFile Repository]
> o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with
> 126 Records and 0 Swap Files in 30 milliseconds (Stop-the-world time = 22
> milliseconds), max Transaction ID 504
> 2026-07-10 00:33:07,417 INFO [Checkpoint FlowFile Repository]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 126 records in 30 milliseconds
> 2026-07-10 00:33:23,751 INFO [Write-Ahead Local State Provider Maintenance]
> o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 2
> Records and 0 Swap Files in 17 milliseconds (Stop-the-world time = 13
> milliseconds), max Transaction ID 4
> 2026-07-10 00:33:25,165 INFO [FileSystemRepository Workers Thread-4]
> o.a.n.c.repository.FileSystemRepository Truncating
> StandardResourceClaim[id=1783657966517-22, container=default, section=22] to
> 20480 bytes because the FlowFile occupying the last 3145728 bytes has been
> removed
> ...
> 2026-07-10 00:44:01,738 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.r.StandardProcessSession Unable to find content for
> StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480,
> length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
> dropping FlowFile
> org.apache.nifi.controller.repository.ContentNotFoundException: Could not
> find content for StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480, length=3145728]: Stream contained only 0 bytes but
> should have contained 3145728
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:99)
> at
> org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
> at
> org.apache.nifi.processors.standard.hash.HashService.traditionalHashStreaming(HashService.java:201)
> at
> org.apache.nifi.processors.standard.hash.HashService.hashValueStreaming(HashService.java:113)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.lambda$onTrigger$0(CryptographicHashContent.java:132)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2738)
> at
> org.apache.nifi.controller.repository.WeakHashMapProcessSessionFactory$FactoryRetainingProcessSession.read(WeakHashMapProcessSessionFactory.java:260)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.onTrigger(CryptographicHashContent.java:132)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1292)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at
> org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> 2026-07-10 00:44:01,739 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.s.CryptographicHashContent
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] Processing
> halted: yielding [1 sec]
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for
> StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480,
> length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
> dropping FlowFile
> at
> org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3756)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2753)
> at
> org.apache.nifi.controller.repository.WeakHashMapProcessSessionFactory$FactoryRetainingProcessSession.read(WeakHashMapProcessSessionFactory.java:260)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.onTrigger(CryptographicHashContent.java:132)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1292)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at
> org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: org.apache.nifi.controller.repository.ContentNotFoundException:
> Could not find content for StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480, length=3145728]: Stream contained only 0 bytes but
> should have contained 3145728
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:99)
> at
> org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
> at
> org.apache.nifi.processors.standard.hash.HashService.traditionalHashStreaming(HashService.java:201)
> at
> org.apache.nifi.processors.standard.hash.HashService.hashValueStreaming(HashService.java:113)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.lambda$onTrigger$0(CryptographicHashContent.java:132)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2738)
> ... 13 common frames omitted
> 2026-07-10 00:44:01,739 WARN [Timer-Driven Process Thread-7]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component
> [CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa]]
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for
> StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480,
> length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
> dropping FlowFile{code}
>
> Interestingly if you replace the nearly empty ExecuteGroovyScript processor
> with an empty UpdateAttribute processor that terminates the success
> relationship, the problem does not occur. I have not done much testing to
> see which processors can cause it, or examined what might be different about
> these two processors that could cause the problem.
> Disabling content claim truncation by setting
> {{nifi.content.claim.truncation.enabled=false}} in nifi.properties prevents
> this problem from occurring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
