Dear Ignite experts, could you please clarify the root cause of the
exception below?
Ignite Version: 2.8.0
Cache configuration (of the affected cache): ATOMIC, PRIMARY_SYNC,
backups=0
Cluster: 4 grid nodes, each 8 cores and 16G heap
E 200526 130653.710 [sys-#48] GridTaskWorker - Failed to obtain remote job
result policy for result from ComputeTask.result(..) method (will fail the
whole task): GridJobResultImpl [job=C4
[r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5c699286],
sib=GridJobSiblingImpl
[sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
nodeId=e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, timeoutObj=null,
attrs=HashMap {}], node=TcpDiscoveryNode
[id=e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, consistentId=10.157.172.70:7700,
addrs=ArrayList [10.157.172.70], sockAddrs=HashSet
[h0010358.associatesys.local/10.157.172.70:7700], discPort=7700, order=2,
intOrder=2, lastExchangeTime=1590516191104, loc=false,
ver=2.8.0#20200226-sha1:341b01df, isClient=false], ex=class
o.a.i.IgniteException: Failed to serialize job exception
[nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, ses=GridJobSessionImpl
[ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4
[r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56],
msg="Failed to marshal object with optimized marshaller: class
o.a.i.compute.ComputeUserUndeclaredException: Failed to execute job due to
unexpected runtime exception
[jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to
update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"],
hasRes=true, isCancelled=false, isOccupied=true]. class
org.apache.ignite.IgniteException: Remote job threw user exception
(override or implement ComputeTask.result(..) method if you would like to
have automatic failover for this exception): Failed to serialize job
exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4
[r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56],
msg="Failed to marshal object with optimized marshaller: class
org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute
job due to unexpected runtime exception
[jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to
update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"]
at
org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102)
at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062)
at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1370)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Failed to serialize job
exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4
[r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56],
msg="Failed to marshal object with optimized marshaller: class
org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute
job due to unexpected runtime exception
[jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to
update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"]
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:887)
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:789)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:639)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1345)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089)
... 7 more
E 200526 130653.712 [OrchestratorPool-#69] GridJobWorker - Failed to
execute job [jobId=dbd07225271-88891f13-2362-424d-9088-f5bb0cb3c9ab,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.scheduling.BuildAccountMetricsOrchestrator$BuildAccountMetricsForPartition,
dep=LocalDeployment [super=GridDeployment [ts=1590516188157,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=004f6225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.scheduling.BuildAccountMetricsOrchestrator$BuildAccountMetricsForPartition,
sesId=cbd07225271-88891f13-2362-424d-9088-f5bb0cb3c9ab,
startTime=1590516385063, endTime=9223372036854775807,
taskNodeId=88891f13-2362-424d-9088-f5bb0cb3c9ab,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=88891f13-2362-424d-9088-f5bb0cb3c9ab,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1645668758]], execName=OrchestratorPool],
jobId=dbd07225271-88891f13-2362-424d-9088-f5bb0cb3c9ab]]. class
org.apache.ignite.IgniteException: Remote job threw user exception
(override or implement ComputeTask.result(..) method if you would like to
have automatic failover for this exception): Failed to serialize job
exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4
[r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56],
msg="Failed to marshal object with optimized marshaller: class
org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute
job due to unexpected runtime exception
[jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to
update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"]
at
org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102)
at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062)
at
org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1370)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Failed to serialize job
exception [nodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, job=C4
[r=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter@5ed7fc56],
msg="Failed to marshal object with optimized marshaller: class
org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to execute
job due to unexpected runtime exception
[jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516189835,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=ed9f6225271-e3088a80-cc76-4f52-ac5e-4f9a4dcf20ef, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=f08cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516413658, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=815042331]], execName=SecondLevelPool],
jobId=b18cd225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to
update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=26504915, instrumentId=26504915}]]"]
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:887)
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:789)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:639)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1345)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089)
... 7 more
E 200526 130654.266 [SecondLevelPool-#115] GridDhtAtomicCache -
<RmInstrument> Unexpected exception during cache update.
java.util.ConcurrentModificationException
at
java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
at java.util.AbstractMap.toString(AbstractMap.java:554)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryPartitionRecovery.collectEntries(CacheContinuousQueryPartitionRecovery.java:261)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.handleEvent(CacheContinuousQueryHandler.java:973)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1019)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2489)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2636)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2097)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1914)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1714)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:486)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:446)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1164)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:624)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2575)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2552)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1302)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856)
at
com.example.app.persistence.dao.cache.CacheRmInstrumentDao.save(CacheRmInstrumentDao.java:35)
at
com.example.app.task.instrument.InstrumentInitializationTask.performInitialization(InstrumentInitializationTask.java:162)
at
com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:95)
at
com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:65)
at java.lang.Iterable.forEach(Iterable.java:75)
at
java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
at
com.example.app.task.instrument.BulkInstrumentInitializationTask.bulkManageInstruments(BulkInstrumentInitializationTask.java:38)
at
com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter.run(BulkInstrumentInitializationTask.java:85)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1944)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:576)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:570)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
E 200526 130654.273 [SecondLevelPool-#115] GridJobWorker - Failed to
execute job due to unexpected runtime exception
[jobId=aad0e225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
dep=LocalDeployment [super=GridDeployment [ts=1590516188157,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1698c449,
clsLdrId=004f6225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter,
sesId=9ad0e225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
startTime=1590516414246, endTime=9223372036854775807,
taskNodeId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
clsLdr=sun.misc.Launcher$AppClassLoader@1698c449, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=null, subjId=37a9d1f5-fb63-4b1a-b220-6deac61d8b7a,
mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1920030771]], execName=SecondLevelPool],
jobId=aad0e225271-37a9d1f5-fb63-4b1a-b220-6deac61d8b7a], err=Failed to
update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=6768608, instrumentId=6768608}]].
org.apache.ignite.cache.CachePartialUpdateException: Failed to update keys
(retry update if possible).: [CacheInstrumentKey{underlyingId=6768608,
instrumentId=6768608}]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1299)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1305)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856)
at
com.example.app.persistence.dao.cache.CacheRmInstrumentDao.save(CacheRmInstrumentDao.java:35)
at
com.example.app.task.instrument.InstrumentInitializationTask.performInitialization(InstrumentInitializationTask.java:162)
at
com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:95)
at
com.example.app.task.instrument.InstrumentInitializationTask.initializeInstrument(InstrumentInitializationTask.java:65)
at java.lang.Iterable.forEach(Iterable.java:75)
at
java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
at
com.example.app.task.instrument.BulkInstrumentInitializationTask.bulkManageInstruments(BulkInstrumentInitializationTask.java:38)
at
com.example.app.task.instrument.BulkInstrumentInitializationTask$IgniteRunnableAdapter.run(BulkInstrumentInitializationTask.java:85)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1944)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:576)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:570)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:499)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class
org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException:
Failed to update keys (retry update if possible).:
[CacheInstrumentKey{underlyingId=6768608, instrumentId=6768608}]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.onPrimaryError(GridNearAtomicAbstractUpdateFuture.java:397)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onPrimaryResponse(GridNearAtomicSingleUpdateFuture.java:253)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture$1.apply(GridNearAtomicAbstractUpdateFuture.java:304)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture$1.apply(GridNearAtomicAbstractUpdateFuture.java:301)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:2001)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1714)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:486)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:446)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1164)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:624)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2575)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2552)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1302)
... 18 more
Suppressed: class org.apache.ignite.IgniteCheckedException:
Failed to update keys on primary node.
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:340)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1999)
... 28 more
Suppressed:
java.util.ConcurrentModificationException
at
java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
at
java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
at
java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
at
java.util.AbstractMap.toString(AbstractMap.java:554)
at
java.lang.String.valueOf(String.java:2994)
at
java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryPartitionRecovery.collectEntries(CacheContinuousQueryPartitionRecovery.java:261)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.handleEvent(CacheContinuousQueryHandler.java:973)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1019)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2489)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2636)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2097)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1914)
... 28 more
Best regards,
Ivan Fedorenkov
