Dear Team,
We have requirement to have multiple caches each node ( 4 nodes) which
will add dynamically . Initially it was working fine . After adding 1000 caches
on each node we are getting memory exception While adding cache to each node
and client node is getting terminated, even though we are increasing the memory
for the client node
Initially we have given memory as -Xmx4G and we have increased the memory to
-Xmx5G but we are getting different error
After Restarting the client node again , the cache is adding to all the nodes.
Please find the logs
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
run.sh: line 20: 32332 Killed java $JVMARGS
[16:50:10,674][SEVERE][grid-timeout-worker-#39%MATCHERWORKER%][G] Blocked
system-critical thread has been detected. This can lead to cluster-wide
undefined behaviour [workerName=partition-exchanger,
threadName=exchange-worker-#65%MATCHERWORKER%, blockedFor=10s]
[16:50:10] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=MATCHERWORKER, finished=false, heartbeatTs=1653045600445]]]
[16:50:22,704][SEVERE][tcp-disco-msg-worker-[29076e7e XX.XX.XXX.IP3:47500
crd]-#2%MATCHERWORKER%][G] Blocked system-critical thread has been detected.
This can lead to cluster-wide undefined behaviour
[workerName=partition-exchanger, threadName=exchange-worker-#65%MATCHERWORKER%,
blockedFor=22s]
[16:50:22] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=partition-exchanger,
igniteInstanceName=MATCHERWORKER, finished=false, heartbeatTs=1653045600445]]]
[16:50:26] Topology snapshot [ver=15, locNode=d38fead7, servers=4, clients=1,
state=ACTIVE, CPUs=64, offheap=9.3GB, heap=25.0GB]
[16:50:26] ^-- Baseline [id=0, size=4, online=4, offline=0]
[ERROR] 2022-05-20 16:50:26.097 [pub-#18934%MATCHERWORKER%] CreateCollectionJob
- RollBack Happened for entityId::726_545_777_4354 for Action::ADDENTITY
[ERROR] 2022-05-20 16:50:26.097 [pub-#18936%MATCHERWORKER%] CreateCollectionJob
- RollBack Happened for entityId::726_545_777_4354 for Action::ADDENTITY
[ERROR] 2022-05-20 16:50:26.097 [pub-#18941%MATCHERWORKER%] CreateCollectionJob
- RollBack Happened for entityId::726_545_777_4354 for Action::ADDENTITY
[ERROR] 2022-05-20 16:50:26.097 [pub-#18937%MATCHERWORKER%] CreateCollectionJob
- RollBack Happened for entityId::726_545_777_4354 for Action::ADDMNT
[ERROR] 2022-05-20 16:50:26.098 [pub-#18938%MATCHERWORKER%] CreateCollectionJob
- RollBack Happened for entityId::726_545_777_4354 for Action::ADDENTITY
[ERROR] 2022-05-20 16:50:26.101 [pub-#18934%MATCHERWORKER%] CreateCollectionJob
- class org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
Got interrupted while waiting for future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
~[ignite-core-2.8.1.jar:2.8.1]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
[Matcher-2.0.21.jar:?]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
[Matcher-2.0.21.jar:?]
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
~[ignite-core-2.8.1.jar:2.8.1]
... 19 more
[16:50:26] Topology snapshot [ver=16, locNode=d38fead7, servers=4, clients=0,
state=ACTIVE, CPUs=64, offheap=9.3GB, heap=20.0GB]
[16:50:26] ^-- Baseline [id=0, size=4, online=4, offline=0]
[ERROR] 2022-05-20 16:50:26.101 [pub-#18938%MATCHERWORKER%] CreateCollectionJob
- class org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
Got interrupted while waiting for future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
~[ignite-core-2.8.1.jar:2.8.1]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
[Matcher-2.0.21.jar:?]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
[Matcher-2.0.21.jar:?]
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
~[ignite-core-2.8.1.jar:2.8.1]
... 19 more
[ERROR] 2022-05-20 16:50:26.103 [pub-#18936%MATCHERWORKER%] CreateCollectionJob
- class org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
Got interrupted while waiting for future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
~[ignite-core-2.8.1.jar:2.8.1]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
[Matcher-2.0.21.jar:?]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
[Matcher-2.0.21.jar:?]
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
~[ignite-core-2.8.1.jar:2.8.1]
... 19 more
[16:50:26,103][SEVERE][pub-#18934%MATCHERWORKER%][GridJobWorker] Failed to
execute job [jobId=6fc2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=grid.cache.CreateCollectionTaskIgnite, dep=LocalDeployment
[super=GridDeployment [ts=1652856068588, depMode=SHARED,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057,
clsLdrId=f7f06e5d081-d38fead7-7569-4a4e-9136-b71be2d10bf4, userVer=0, loc=true,
sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false,
usage=0]], taskClsName=grid.cache.CreateCollectionTaskIgnite,
sesId=aec2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
startTime=1653045584321, endTime=9223372036854775807,
taskNodeId=4df513fd-444c-4931-b835-be3914bcdd7b,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=o.a.i.i.cluster.ClusterGroupAdapter$AttributeFilter@59859b0e,
subjId=4df513fd-444c-4931-b835-be3914bcdd7b, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
hash=1642816517]], execName=null],
jobId=6fc2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b]]
class org.apache.ignite.IgniteException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:179)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
... 15 more
Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted
while waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
... 19 more
[16:50:26,104][SEVERE][pub-#18936%MATCHERWORKER%][GridJobWorker] Failed to
execute job [jobId=efc2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=grid.cache.CreateCollectionTaskIgnite, dep=LocalDeployment
[super=GridDeployment [ts=1652856068588, depMode=SHARED,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057,
clsLdrId=f7f06e5d081-d38fead7-7569-4a4e-9136-b71be2d10bf4, userVer=0, loc=true,
sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false,
usage=0]], taskClsName=grid.cache.CreateCollectionTaskIgnite,
sesId=aec2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
startTime=1653045584321, endTime=9223372036854775807,
taskNodeId=4df513fd-444c-4931-b835-be3914bcdd7b,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=o.a.i.i.cluster.ClusterGroupAdapter$AttributeFilter@14c8112f,
subjId=4df513fd-444c-4931-b835-be3914bcdd7b, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
hash=1402396768]], execName=null],
jobId=efc2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b]]
class org.apache.ignite.IgniteException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:179)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
... 15 more
Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted
while waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
... 19 more
[ERROR] 2022-05-20 16:50:26.103 [pub-#18941%MATCHERWORKER%] CreateCollectionJob
- class org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
Got interrupted while waiting for future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
~[ignite-core-2.8.1.jar:2.8.1]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
[Matcher-2.0.21.jar:?]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
[Matcher-2.0.21.jar:?]
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
~[ignite-core-2.8.1.jar:2.8.1]
... 19 more
[ERROR] 2022-05-20 16:50:26.103 [pub-#18937%MATCHERWORKER%] CreateCollectionJob
- class org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
Got interrupted while waiting for future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
~[ignite-core-2.8.1.jar:2.8.1]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:155)
[Matcher-2.0.21.jar:?]
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
[Matcher-2.0.21.jar:?]
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
[ignite-core-2.8.1.jar:2.8.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.IgniteInterruptedException: Got interrupted while
waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
~[ignite-core-2.8.1.jar:2.8.1]
... 19 more
[16:50:26,104][SEVERE][pub-#18938%MATCHERWORKER%][GridJobWorker] Failed to
execute job [jobId=60d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=grid.cache.CreateCollectionTaskIgnite, dep=LocalDeployment
[super=GridDeployment [ts=1652856068588, depMode=SHARED,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057,
clsLdrId=f7f06e5d081-d38fead7-7569-4a4e-9136-b71be2d10bf4, userVer=0, loc=true,
sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false,
usage=0]], taskClsName=grid.cache.CreateCollectionTaskIgnite,
sesId=aec2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
startTime=1653045584321, endTime=9223372036854775807,
taskNodeId=4df513fd-444c-4931-b835-be3914bcdd7b,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=o.a.i.i.cluster.ClusterGroupAdapter$AttributeFilter@5cdc8ffc,
subjId=4df513fd-444c-4931-b835-be3914bcdd7b, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
hash=1456209804]], execName=null],
jobId=60d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b]]
class org.apache.ignite.IgniteException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:179)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
... 15 more
Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted
while waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
... 19 more
[16:50:26,105][SEVERE][pub-#18941%MATCHERWORKER%][GridJobWorker] Failed to
execute job [jobId=a1d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=grid.cache.CreateCollectionTaskIgnite, dep=LocalDeployment
[super=GridDeployment [ts=1652856068588, depMode=SHARED,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057,
clsLdrId=f7f06e5d081-d38fead7-7569-4a4e-9136-b71be2d10bf4, userVer=0, loc=true,
sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false,
usage=0]], taskClsName=grid.cache.CreateCollectionTaskIgnite,
sesId=aec2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
startTime=1653045584321, endTime=9223372036854775807,
taskNodeId=4df513fd-444c-4931-b835-be3914bcdd7b,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=o.a.i.i.cluster.ClusterGroupAdapter$AttributeFilter@472f7f41,
subjId=4df513fd-444c-4931-b835-be3914bcdd7b, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
hash=121825609]], execName=null],
jobId=a1d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b]]
class org.apache.ignite.IgniteException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:179)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:131)
... 15 more
Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted
while waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
... 19 more
[16:50:26,105][SEVERE][pub-#18937%MATCHERWORKER%][GridJobWorker] Failed to
execute job [jobId=00d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=grid.cache.CreateCollectionTaskIgnite, dep=LocalDeployment
[super=GridDeployment [ts=1652856068588, depMode=SHARED,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057,
clsLdrId=f7f06e5d081-d38fead7-7569-4a4e-9136-b71be2d10bf4, userVer=0, loc=true,
sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false,
usage=0]], taskClsName=grid.cache.CreateCollectionTaskIgnite,
sesId=aec2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
startTime=1653045584321, endTime=9223372036854775807,
taskNodeId=4df513fd-444c-4931-b835-be3914bcdd7b,
clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
topPred=o.a.i.i.cluster.ClusterGroupAdapter$AttributeFilter@1f641fbb,
subjId=4df513fd-444c-4931-b835-be3914bcdd7b, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
hash=1325084889]], execName=null],
jobId=00d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b]]
class org.apache.ignite.IgniteException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:179)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:31)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:596)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7005)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:590)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for
future to complete.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3321)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3285)
at grid.cache.CreateCollectionJob.execute(CreateCollectionJob.java:155)
... 15 more
Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted
while waiting for future to complete.
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
at
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
... 19 more
[16:50:26,125][SEVERE][pub-#18939%MATCHERWORKER%][TcpCommunicationSpi] Failed
to send message to remote node [node=TcpDiscoveryNode
[id=4df513fd-444c-4931-b835-be3914bcdd7b,
consistentId=4df513fd-444c-4931-b835-be3914bcdd7b, addrs=ArrayList
[XX.XX.XXX.IP1, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:0,
IP1/XX.XX.XXX.IP1:0], discPort=0, order=13, intOrder=9,
lastExchangeTime=1653041973282, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_TASK, topicOrd=2,
ordered=false, timeout=0, skipOnTimeout=false, msg=GridJobExecuteResponse
[nodeId=d38fead7-7569-4a4e-9136-b71be2d10bf4,
sesId=aec2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b,
jobId=e0d2bf0e081-4df513fd-444c-4931-b835-be3914bcdd7b, gridEx=null,
isCancelled=false, retry=null]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): TcpDiscoveryNode
[id=4df513fd-444c-4931-b835-be3914bcdd7b,
consistentId=4df513fd-444c-4931-b835-be3914bcdd7b, addrs=ArrayList
[XX.XX.XXX.IP1, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:0,
IP1/XX.XX.XXX.IP1:0], discPort=0, order=13, intOrder=9,
lastExchangeTime=1653041973282, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3539)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2132)
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:964)
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:809)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:659)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:519)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1367)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2121)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[16:50:26,137][SEVERE][exchange-worker-#65%MATCHERWORKER%][TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=cbaae4af-68aa-4a15-b3f6-6c658941c03b,
consistentId=cbaae4af-68aa-4a15-b3f6-6c658941c03b, addrs=ArrayList
[XX.XX.XXX.IP1, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:0,
IP1/XX.XX.XXX.IP1:0], discPort=0, order=14, intOrder=10,
lastExchangeTime=1653041980832, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsFullMessage
[parts=HashMap {735046623=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335,
size=261], d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645,
size=258], 29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335,
size=236], 29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=269]}, -1048153706=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335,
size=518], d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645,
size=504], 29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335,
size=496], 29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=530]}, 1221286107=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, -1048153705=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, 1221286106=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, -1048153708=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, 1221286105=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, -1048153707=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, 1547508782=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, 1221286104=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, 1221286108=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, 1221286099=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=334,
size=0]}, -1048153700=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4]Ver=14, minorTopVer=4],
updateSeq=645, size=0]... and 133667 skipped ...0]},
1862323742=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335,
size=0]}... and 1441 more}, partCntrs=null,
partCntrs2=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap2@4e32d588,
partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [],
partsToReload=IgniteDhtPartitionsToReloadMap [], topVer=AffinityTopologyVersion
[topVer=14, minorTopVer=4], errs=null, resTopVer=null, flags=1, partCnt=1541,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=14, minorTopVer=4],
discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=14, minorTopVer=4],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=d38fead7-7569-4a4e-9136-b71be2d10bf4, consistentId=matcher_204,
addrs=ArrayList [XX.XX.XXX.IP1, 127.0.0.1], sockAddrs=HashSet
[/127.0.0.1:47500, IP1/XX.XX.XXX.IP1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1653045626109, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=false], topVer=14, nodeId8=d38fead7,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1653045584361]], nodeId=d38fead7,
evt=DISCOVERY_CUSTOM_EVT], lastVer=GridCacheVersion [topVer=264336153,
order=1652856147134, nodeOrder=1], super=GridCacheMessage [msgId=278,
depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=14,
minorTopVer=4], err=null, skipPrepare=false]]]... and 133668 skipped ...]},
1862323742=GridDhtPartitionFullMap
{a57747be-1ab9-4d59-8a24-49a3f6b4f4fe=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
d38fead7-7569-4a4e-9136-b71be2d10bf4=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=645, size=0],
29076e7e-07c5-4498-9630-4f3d1c0cc15f=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335, size=0],
29d41148-4790-4807-a375-00853769a509=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=14, minorTopVer=4], updateSeq=335,
size=0]}... and 1441 more}, partCntrs=null,
partCntrs2=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap2@4e32d588,
partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [],
partsToReload=IgniteDhtPartitionsToReloadMap [], topVer=AffinityTopologyVersion
[topVer=14, minorTopVer=4], errs=null, resTopVer=null, flags=1, partCnt=1541,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=14, minorTopVer=4],
discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=14, minorTopVer=4],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=d38fead7-7569-4a4e-9136-b71be2d10bf4, consistentId=matcher_204,
addrs=ArrayList [XX.XX.XXX.IP1, 127.0.0.1], sockAddrs=HashSet
[/127.0.0.1:47500, IP1/XX.XX.XXX.IP1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1653045626109, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=false], topVer=14, nodeId8=d38fead7,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1653045584361]], nodeId=d38fead7,
evt=DISCOVERY_CUSTOM_EVT], lastVer=GridCacheVersion [topVer=264336153,
order=1652856147134, nodeOrder=1], super=GridCacheMessage [msgId=278,
depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=14,
minorTopVer=4], err=null, skipPrepare=false]]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException:
Failed to send message (node left topology): TcpDiscoveryNode
[id=cbaae4af-68aa-4a15-b3f6-6c658941c03b,
consistentId=cbaae4af-68aa-4a15-b3f6-6c658941c03b, addrs=ArrayList
[XX.XX.XXX.IP1, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:0,
IP1/XX.XX.XXX.IP1:0], discPort=0, order=14, intOrder=10,
lastExchangeTime=1653041980832, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3539)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$6800(TcpCommunicationSpi.java:287)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4913)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4714)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2423)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)[14:24:43,217][SEVERE][tcp-client-disco-sock-writer-#6%TTMASTER%][TcpDiscoverySpi]
Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=963bd9fe081-b44ff99d-5d37-4827-9386-6d1837dfc1e6,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1417)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[INFO ] 2022-05-23 14:24:44.390 [sys-#254%MATCHERMASTER%]
CreateCollectionTaskIgnite - came to result policy
[INFO ] 2022-05-23 14:25:00.981 [sys-#266%MATCHERMASTER%]
CreateCollectionTaskIgnite - came to result policy
[14:25:03,341][SEVERE][tcp-client-disco-sock-writer-#2%MATCHERMASTER%][TcpDiscoverySpi]
Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=473bd9fe081-b3e56a48-e00b-4c3b-a701-2ebe79e59afe,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1417)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[INFO ] 2022-05-23 14:25:17.817 [sys-#264%MATCHERMASTER%]
CreateCollectionTaskIgnite - came to result policy
[14:25:17,821][SEVERE][sys-#267%MATCHERMASTER%][GridCacheIoManager] Failed to
process message [senderId=57dc6e6c-723d-4738-ad66-efb90b2c28ac,
msg=GridDhtPartitionsFullMessage [parts=null, partCntrs=null, partCntrs2=null,
partHistSuppliers=null, partsToReload=null, topVer=AffinityTopologyVersion
[topVer=57, minorTopVer=171], errs=null, resTopVer=null, flags=1, partCnt=0,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=57, minorTopVer=171], discoEvt=null,
nodeId=8ba33123, evt=DISCOVERY_CUSTOM_EVT], lastVer=GridCacheVersion [topVer=0,
order=1653287141951, nodeOrder=0], super=GridCacheMessage [msgId=4237,
depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=57,
minorTopVer=171], err=null, skipPrepare=false]]]]
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:116)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshalZip(IgniteUtils.java:10248)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:546)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:543)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11157)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11059)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage.finishUnmarshal(GridDhtPartitionsFullMessage.java:539)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1625)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[14:25:17,833][SEVERE][sys-#267%MATCHERMASTER%][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError:
Java heap space]]
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:116)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshalZip(IgniteUtils.java:10248)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:546)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:543)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11157)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11059)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage.finishUnmarshal(GridDhtPartitionsFullMessage.java:539)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1625)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[14:25:20,170][SEVERE][sys-#258%TTMASTER%][GridCacheIoManager] Failed to
process message [senderId=57dc6e6c-723d-4738-ad66-efb90b2c28ac,
msg=GridDhtPartitionsFullMessage [parts=null, partCntrs=null, partCntrs2=null,
partHistSuppliers=null, partsToReload=null, topVer=AffinityTopologyVersion
[topVer=57, minorTopVer=171], errs=null, resTopVer=null, flags=1, partCnt=0,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=57, minorTopVer=171], discoEvt=null,
nodeId=8ba33123, evt=DISCOVERY_CUSTOM_EVT], lastVer=GridCacheVersion [topVer=0,
order=1653287141951, nodeOrder=0], super=GridCacheMessage [msgId=4239,
depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=57,
minorTopVer=171], err=null, skipPrepare=false]]]]
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:116)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshalZip(IgniteUtils.java:10248)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloaderpreloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:546)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:543)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11157)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11059)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage.finishUnmarshal(GridDhtPartitionsFullMessage.java:539)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1625)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[14:25:20,171][SEVERE][sys-#258%TTMASTER%][] Critical system error detected.
Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError:
Java heap space]]
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:116)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshalZip(IgniteUtils.java:10248)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:546)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage$2.apply(GridDhtPartitionsFullMessage.java:543)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11157)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11059)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage.finishUnmarshal(GridDhtPartitionsFullMessage.java:539)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1625)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[14:25:20,170][SEVERE][tcp-client-disco-sock-writer-#2%MATCHERMASTER%][TcpDiscoverySpi]
Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=3d3bd9fe081-b3e56a48-e00b-4c3b-a701-2ebe79e59afe,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1417)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[14:25:23,736][SEVERE][sys-#267%MATCHERMASTER%][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: Java heap space]]
[14:25:24,944][SEVERE][sys-#258%TTMASTER%][] JVM will be halted immediately due
to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.OutOfMemoryError: Java heap space]]
