Dear Ignite experts, could you please clarify the root cause of the exception below?
Ignite Version: 2.8.0 Cache configuration (of the affected cache): ATOMIC, PRIMARY_SYNC, backups=0 Cluster: 4 grid nodes, each 8 cores and 16G heap E 200526 130653.710 [sys-#48] GridTaskWorker - Failed to obtain remote job result policy for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl [job=C4 [r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5c699286], sib=GridJobSiblingImpl [sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, nodeId=e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, isJobDone=false], jobCtx=GridJobContextImpl [jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, timeoutObj=null, attrs=HashMap {}], node=TcpDiscoveryNode [id=e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, consistentId=10.157.172.70:7700, addrs=ArrayList [10.157.172.70], sockAddrs=HashSet [h0010358.associatesys.local/10.157.172.70:7700], discPort=7700, order=2, intOrder=2, lastExchangeTime=1590516191104, loc=false, ver=2.8.0#20200226-sha1:341b01df, isClient=false], ex=class o.a.i.IgniteException: Failed to serialize job exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4 [r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56], msg="Failed to marshal object with optimized marshaller: class o.a.i.compute.ComputeUserUndeclaredException: Failed to execute job due to unexpected runtime exception [jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"], hasRes=true, isCancelled=false, isOccupied=true]. class org.apache.ignite.IgniteException: Remote job threw user exception (override or implement ComputeTask.result(..) method if you would like to have automatic failover for this exception): Failed to serialize job exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4 [r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56], msg="Failed to marshal object with optimized marshaller: class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute job due to unexpected runtime exception [jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"] at org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102) at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062) at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037) at org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055) at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862) at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146) at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1370) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468) at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.IgniteException: Failed to serialize job exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4 [r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56], msg="Failed to marshal object with optimized marshaller: class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute job due to unexpected runtime exception [jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"] at org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:887) at org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:789) at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:639) at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1345) at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089) ... 7 more E 200526 130653.712 [OrchestratorPool-#69] GridJobWorker - Failed to execute job [jobId=dbd07225271-88891f13-2362-424d-9088-f5bb0cb3c9ab, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.scheduling.BuildAccountMetricsOrchestrator$BuildAccountMetricsForPartition, dep=LocalDeployment [super=GridDeployment [ts=1590516188157, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=004f6225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.scheduling.BuildAccountMetricsOrchestrator$BuildAccountMetricsForPartition, sesId=cbd07225271-88891f13-2362-424d-9088-f5bb0cb3c9ab, startTime=1590516385063, endTime=9223372036854775807, taskNodeId=88891f13-2362-424d-9088-f5bb0cb3c9ab, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=88891f13-2362-424d-9088-f5bb0cb3c9ab, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1645668758]], execName=OrchestratorPool], jobId=dbd07225271-88891f13-2362-424d-9088-f5bb0cb3c9ab]]. class org.apache.ignite.IgniteException: Remote job threw user exception (override or implement ComputeTask.result(..) method if you would like to have automatic failover for this exception): Failed to serialize job exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4 [r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56], msg="Failed to marshal object with optimized marshaller: class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute job due to unexpected runtime exception [jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"] at org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102) at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062) at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037) at org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055) at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862) at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146) at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1370) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468) at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.IgniteException: Failed to serialize job exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4 [r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56], msg="Failed to marshal object with optimized marshaller: class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute job due to unexpected runtime exception [jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516189835, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516413658, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=815042331]], execName=SecondLevelPool], jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"] at org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:887) at org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:789) at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:639) at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1345) at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089) ... 7 more E 200526 130654.266 [SecondLevelPool-#115] GridDhtAtomicCache - <RmInstrument> Unexpected exception during cache update. java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242) at java.util.AbstractMap.toString(AbstractMap.java:554) at java.lang.String.valueOf(String.java:2994) at java.lang.StringBuilder.append(StringBuilder.java:131) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryPartitionRecovery.collectEntries(CacheContinuousQueryPartitionRecovery.java:261) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.handleEvent(CacheContinuousQueryHandler.java:973) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1019) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2489) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2636) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2097) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1914) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1714) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:486) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:446) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1164) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:624) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2575) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2552) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1302) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856) at com.example.app.persistence.dao.cache.CacheRmInstrumentDao.save(CacheRmInstrumentDao.java:35) at com.example.app.task.instrument.InstrumentInitializationTask.performInitialization(InstrumentInitializationTask.java:162) at com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:95) at com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:65) at java.lang.Iterable.forEach(Iterable.java:75) at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) at com.example.app.task.instrument.BulkInstrumentInitializationTask.bulkManageInstruments(BulkInstrumentInitializationTask.java:38) at com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter.run(BulkInstrumentInitializationTask.java:85) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1944) at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:576) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005) at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:570) at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) E 200526 130654.273 [SecondLevelPool-#115] GridJobWorker - Failed to execute job due to unexpected runtime exception [jobId=aad0e225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, dep=LocalDeployment [super=GridDeployment [ts=1590516188157, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, clsLdrId=004f6225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter, sesId=9ad0e225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, startTime=1590516414246, endTime=9223372036854775807, taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1920030771]], execName=SecondLevelPool], jobId=aad0e225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=6768608, instrumentId=6768608}]]. org.apache.ignite.cache.CachePartialUpdateException: Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=6768608, instrumentId=6768608}] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1299) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1305) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856) at com.example.app.persistence.dao.cache.CacheRmInstrumentDao.save(CacheRmInstrumentDao.java:35) at com.example.app.task.instrument.InstrumentInitializationTask.performInitialization(InstrumentInitializationTask.java:162) at com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:95) at com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:65) at java.lang.Iterable.forEach(Iterable.java:75) at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) at com.example.app.task.instrument.BulkInstrumentInitializationTask.bulkManageInstruments(BulkInstrumentInitializationTask.java:38) at com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter.run(BulkInstrumentInitializationTask.java:85) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1944) at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:576) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005) at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:570) at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException: Failed to update keys (retry update if possible).: [CacheInstrumentKey{underlyingId=6768608, instrumentId=6768608}] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.onPrimaryError(GridNearAtomicAbstractUpdateFuture.java:397) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onPrimaryResponse(GridNearAtomicSingleUpdateFuture.java:253) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture$1.apply(GridNearAtomicAbstractUpdateFuture.java:304) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture$1.apply(GridNearAtomicAbstractUpdateFuture.java:301) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:2001) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1714) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:486) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:446) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1164) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:624) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2575) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2552) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1302) ... 18 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node. at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:340) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1999) ... 28 more Suppressed: java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242) at java.util.AbstractMap.toString(AbstractMap.java:554) at java.lang.String.valueOf(String.java:2994) at java.lang.StringBuilder.append(StringBuilder.java:131) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryPartitionRecovery.collectEntries(CacheContinuousQueryPartitionRecovery.java:261) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.handleEvent(CacheContinuousQueryHandler.java:973) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1019) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2489) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2636) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2097) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1914) ... 28 more Best regards, Ivan Fedorenkov