[
https://issues.apache.org/jira/browse/NIFI-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18019907#comment-18019907
]
Peter Gyori commented on NIFI-12468:
------------------------------------
Thank you [~shubh_init] for your comment, it helped me reproduce the issue
without using break points.
An important setting is to increase the available memory for the Java process
so that the timeout of the long operation happens sooner than an OOM would. If
only 1 GB of memory is allowed for the process, OOM happens sooner than the
timeout would.
Steps to reproduce:
# Set the NiFi memory to at least 8 GB in bootstrap.conf
# Start NiFi with Java21.
# Perform actions on the canvas (e.g. add and/or delete processors) until 1.8
million action entries get inserted into the flow configuration history. At
that point the database_repository folder consumes about 694 MB disk space.
# View the flow configuration history, click “Purge History” and set a future
end date.
# Wait for about 80 seconds.
A pop-up window will appear. After closing it, neither the processors can be
configured, nor the flow configuration history can be viewed until NiFi is
restarted.
The issue can be reproduced with NiFi 1.28 as well. The interesting thing is
that it happens with Java 21 and Java 17 but not with Java 11.
> Xodus Flow Configuration History Environment Inoperative after Stream Closed
> ----------------------------------------------------------------------------
>
> Key: NIFI-12468
> URL: https://issues.apache.org/jira/browse/NIFI-12468
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.0.0-M1
> Reporter: Michael W Moser
> Priority: Minor
>
> I caused Xodus to timeout writing a transation to flow configuration history.
> Thereafter, every operation that would update flow configuration history, or
> trying to view flow configuration history, continues to throw an exception.
> The UI shows "An unexpected error has occurred. Please check the logs for
> additional details" based on a 500 response from the REST API
> /nifi-api/flow/history
> I had to restart NiFi to recover. Hopefully, this code can be improved to
> detect exceptions and recover on its own.
> I caused the initial problem by setting a breakpoint while a debugger was
> attached to NiFi. I dragged a new Process Group onto the graph from a flow
> definition file and let it sit at the breakpoint for a while.
> {noformat}
> 2023-12-04 16:32:00,448 ERROR [NiFi Web Server-466]
> jetbrains.exodus.env.EnvironmentImpl Failed to flush transaction
> java.io.IOException: Stream Closed
> at java.base/java.io.RandomAccessFile.length0(Native Method)
> at java.base/java.io.RandomAccessFile.length(RandomAccessFile.java:653)
> at jetbrains.exodus.io.FileDataWriter.ensureFile(FileDataWriter.kt:172)
> at jetbrains.exodus.io.FileDataWriter.write(FileDataWriter.kt:59)
> at
> jetbrains.exodus.log.BufferedDataWriter.writePage(BufferedDataWriter.java:264)
> at
> jetbrains.exodus.log.BufferedDataWriter.flush(BufferedDataWriter.java:176)
> at jetbrains.exodus.log.Log.flush(Log.kt:631)
> at jetbrains.exodus.log.Log.flush$default(Log.kt:630)
> at jetbrains.exodus.log.Log.flush(Log.kt)
> at
> jetbrains.exodus.env.EnvironmentImpl.flushTransaction(EnvironmentImpl.java:731)
> at
> jetbrains.exodus.env.EnvironmentImpl.commitTransaction(EnvironmentImpl.java:684)
> at
> jetbrains.exodus.env.ReadWriteTransaction.commit(ReadWriteTransaction.java:96)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.doCommit(PersistentStoreTransaction.java:192)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.commit(PersistentStoreTransaction.java:180)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:720)
> at
> org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116)
> at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66)
> at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> {noformat}
> Afterwards, any attempt to read or write from Xodus produces an exception
> similar to this:
> {noformat}
> 2023-12-04 16:32:53,111 WARN [NiFi Web Server-371]
> o.apache.nifi.audit.ProcessGroupAuditor Unable to record actions:
> jetbrains.exodus.ExodusException: Environment is inoperative
> at
> jetbrains.exodus.ExodusException.toExodusException(ExodusException.java:49)
> at
> jetbrains.exodus.env.EnvironmentImpl.checkIsOperative(EnvironmentImpl.java:1069)
> at
> jetbrains.exodus.env.EnvironmentImpl.beginTransaction(EnvironmentImpl.java:613)
> at
> jetbrains.exodus.env.EnvironmentImpl.beginExclusiveTransaction(EnvironmentImpl.java:264)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.<init>(PersistentStoreTransaction.java:130)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.beginExclusiveTransaction(PersistentEntityStoreImpl.java:539)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:717)
> at
> org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116)
> at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66)
> at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> ...
> ...
> Caused by: java.io.IOException: Stream Closed
> at java.base/java.io.RandomAccessFile.length0(Native Method)
> at java.base/java.io.RandomAccessFile.length(RandomAccessFile.java:653)
> at jetbrains.exodus.io.FileDataWriter.ensureFile(FileDataWriter.kt:172)
> at jetbrains.exodus.io.FileDataWriter.write(FileDataWriter.kt:59)
> at
> jetbrains.exodus.log.BufferedDataWriter.writePage(BufferedDataWriter.java:264)
> at
> jetbrains.exodus.log.BufferedDataWriter.flush(BufferedDataWriter.java:176)
> at jetbrains.exodus.log.Log.flush(Log.kt:631)
> at jetbrains.exodus.log.Log.flush$default(Log.kt:630)
> at jetbrains.exodus.log.Log.flush(Log.kt)
> at
> jetbrains.exodus.env.EnvironmentImpl.flushTransaction(EnvironmentImpl.java:731)
> at
> jetbrains.exodus.env.EnvironmentImpl.commitTransaction(EnvironmentImpl.java:684)
> at
> jetbrains.exodus.env.ReadWriteTransaction.commit(ReadWriteTransaction.java:96)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.doCommit(PersistentStoreTransaction.java:192)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.commit(PersistentStoreTransaction.java:180)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:720)
> at
> org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116)
> at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66)
> at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
