Re: Blocked system-critical due to a listener?
I didn't read your post carefully, sorry. > The problem seems to be a listener that we register with ignite.events(forLocal).remoteListen() that deletes/puts some cache entries during NODE_LEFT events. Are you listening to NODE_LEFT events and then trigger some cache actions (put / get)? сб, 2 апр. 2022 г. в 22:34, Ilya Shishkov : > Hi, > > Could you, please, attach logs from failed nodes? > > ср, 23 мар. 2022 г. в 14:07, Joan Pujol : > >> Hi, >> >> We had a cluster problem while two of the four server nodes left the >> cluster and started again in a cluster running Ignite 2.10 >> The problem seems to be that Partition Exchange didn't finish >> >> In particular, the problem seems to be in one of the remaining nodes that >> has a Blocked system-critical error on disco-event-worker that is locked >> for more than 19s. >> I checked GC logs and there isn't any relevant pause when the block occurs >> >> Some relevant logs: >> >> remaining node: >> >2022-03-22 13:33:42.439 WARN [services-deployment-worker-#95] >> o.a.i.i.p.s.ServiceDeploymentManager - Failed to wait service deployment >> process or timeout had been reached, timeout=1, >> taskDepId=ServiceDeploymentProcessId [topVer=AffinityTopologyVersion >> [topVer=9, minorTopVer=0], reqId=null] >> >2022-03-22 13:33:42.687 WARN [exchange-worker-#54] >> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Unable to await >> partitions release latch within timeout. Some nodes have not sent >> acknowledgement for latch completion. It's possible due to unfinishined >> atomic updates, transactions or not released explicit locks on that nodes. >> Please check logs for errors on nodes with ids reported in latch >> `pendingAcks` collection [latch=ServerLatch [permits=1, >> pendingAcks=HashSet [6d63e1bd-9de4-4341-81bb-c3a293ebe2eb], >> super=CompletableLatch [id=CompletableLatchUid [id=exchange, >> topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0] >> >2022-03-22 13:33:51.681 ERROR [tcp-disco-msg-worker-[5bc334df 0:0:0:0:0: >> 0:0:1%lo:47500]-#2-#45] o.a.i.i.u.typedef.G - Blocked system-critical >> thread has been detected. This can lead to cluster-wide undefined behaviour >> [workerName=disco-event-worker, threadName=disco-event-worker-#50, >> blockedFor=19s] >> >2022-03-22 13:33:51.701 WARN [tcp-disco-msg-worker-[5bc334df 0:0:0:0:0: >> 0:0:1%lo:47500]-#2-#45] o.a.ignite.Ignite - Possible failure suppressed >> accordingly to a configured handler [hnd=AbstractFailureHandler >> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, >> SYSTEM_CRITICAL_OPERATION_TIMEOUT]], failureCtx=FailureContext >> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker >> [name=disco-event-worker, igniteInstanceName=null, finished=false, >> heartbeatTs=1647956012230]]] >> org.apache.ignite.IgniteException: GridWorker [name=disco-event-worker, >> igniteInstanceName=null, finished=false, heartbeatTs=1647956012230] >> at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method) >> at java.base@11.0.9 >> /java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) >> at >> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) >> at >> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) >> at >> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.remove0(GridDhtAtomicCache.java:746) >> >2022-03-22 13:33:51.701 WARN [tcp-disco-msg-worker-[5bc334df 0:0:0:0:0: >> 0:0:1%lo:47500]-#2-#45] o.a.i.i.p.f.FailureProcessor - No deadlocked >> threads detected. >> ... >> A thread dump where the blocked thread thread dump is: >> Thread [name="disco-event-worker-#50", id=306, state=WAITING, blockCnt=1, >> waitCnt=543567] >> at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method) >> at java.base@11.0.9 >> /java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) >> at >> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) >> at >> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) >> at >> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.remove0(GridDhtAtomicCache.java:746) >> at >> o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3397) >> at >> o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3380) >> at >> o.a.i.i.processors.cache.IgniteCacheProxyImpl.remove(IgniteCacheProxyImpl.java:1437) >> at >> o.a.i.i.processors.cache.GatewayProtectedCacheProxy.remove(GatewayProtectedCacheProxy.java:952) >> at >> es.decide.dwm.service.IgniteManager.removeProcessingRequest(IgniteManager.java:186) >> at >> es.decide.dwm.service.IgniteManager.lambda$registerIgniteEvents$1(IgniteManager.java:178) >> at >> es.decide.dwm.service.IgniteManager$$Lambda$8656/0x00080019d040.accept(Unknown >> Source) >> at java.base@11.0.9 >>
Re: Return result stream from ignite compute
Hey,
They are deployed already. What itbis complaining about is the lambda
inside stream.map(). If I remove it, it works fine.
Question is how can we work around stream operations which always take
lambda function.
On Sun, 3 Apr 2022, 00:15 Ilya Shishkov, wrote:
> Hi Surinder,
>
> Your tasks and nested classes should be in the class path of the
> server node. You can deploy them manually or automatically by means of
> peer-class loading [1].
>
> 1.
> https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading
>
> пт, 1 апр. 2022 г. в 15:12, Surinder Mehra :
>
>> Hi, I am trying to return the result stream from ignite compute task.
>> When compute task has a map() on stream it fails with below error . Can
>> someone please explain.
>>
>> "Exception in thread "main" class
>> org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
>> object [typeName=java.util.stream.ReferencePipeline$3]"
>>
>> Setup: One default ignite node
>> Compute task
>> client node to submit compute task and collect the stream results.
>>
>> public class StreamTask implements IgniteCallable> {
>> @Override
>> public Stream call() throws Exception {
>>
>> Function integerDepartmentFunction = i -> new
>> MyFunction().apply(i);
>> return IntStream.of(1, 2, 3).boxed().map(integerDepartmentFunction);
>> }
>> }
>>
>> class MyFunction implements Function {
>> @Override
>> public Department apply(Integer integer) {
>> return new Department(integer, "sdf"+ integer);
>> }
>> }
>>
>>
>> IgniteConfiguration cfg = new IgniteConfiguration();
>> cfg.setClientMode(true);
>> try (Ignite ignite = Ignition.start(cfg)) {
>>
>> ClusterGroup serversGrp = ignite.cluster().forServers();
>>
>> Stream stream = ignite.compute(serversGrp).call(new
>> StreamTask());
>> System.out.println("Stream : "+ stream.collect(Collectors.toList()));
>>
>> }
>>
>> Error:
>>
>> Exception in thread "main" class
>> org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object
>> [typeName=java.util.stream.ReferencePipeline$3]
>> at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:971)
>> at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1769)
>> at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1721)
>> at
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:319)
>> at
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:304)
>> at
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:101)
>> at
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
>> at
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10540)
>>
>> Caused by: java.lang.ClassNotFoundException:
>> training.ignite.compute.StreamTask$$Lambda$894/0x000800670840
>> at java.base/java.lang.Class.forName0(Native Method)
>> at java.base/java.lang.Class.forName(Class.java:398)
>> at
>> org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9064)
>> at
>> org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9002)
>> at
>> org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:376)
>> at
>> org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:693)
>>
>>
Re: Blocked system-critical due to a listener?
Hi, Could you, please, attach logs from failed nodes? ср, 23 мар. 2022 г. в 14:07, Joan Pujol : > Hi, > > We had a cluster problem while two of the four server nodes left the > cluster and started again in a cluster running Ignite 2.10 > The problem seems to be that Partition Exchange didn't finish > > In particular, the problem seems to be in one of the remaining nodes that > has a Blocked system-critical error on disco-event-worker that is locked > for more than 19s. > I checked GC logs and there isn't any relevant pause when the block occurs > > Some relevant logs: > > remaining node: > >2022-03-22 13:33:42.439 WARN [services-deployment-worker-#95] > o.a.i.i.p.s.ServiceDeploymentManager - Failed to wait service deployment > process or timeout had been reached, timeout=1, > taskDepId=ServiceDeploymentProcessId [topVer=AffinityTopologyVersion > [topVer=9, minorTopVer=0], reqId=null] > >2022-03-22 13:33:42.687 WARN [exchange-worker-#54] > o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Unable to await > partitions release latch within timeout. Some nodes have not sent > acknowledgement for latch completion. It's possible due to unfinishined > atomic updates, transactions or not released explicit locks on that nodes. > Please check logs for errors on nodes with ids reported in latch > `pendingAcks` collection [latch=ServerLatch [permits=1, > pendingAcks=HashSet [6d63e1bd-9de4-4341-81bb-c3a293ebe2eb], > super=CompletableLatch [id=CompletableLatchUid [id=exchange, > topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0] > >2022-03-22 13:33:51.681 ERROR [tcp-disco-msg-worker-[5bc334df 0:0:0:0:0:0 > :0:1%lo:47500]-#2-#45] o.a.i.i.u.typedef.G - Blocked system-critical > thread has been detected. This can lead to cluster-wide undefined behaviour > [workerName=disco-event-worker, threadName=disco-event-worker-#50, > blockedFor=19s] > >2022-03-22 13:33:51.701 WARN [tcp-disco-msg-worker-[5bc334df 0:0:0:0:0:0 > :0:1%lo:47500]-#2-#45] o.a.ignite.Ignite - Possible failure suppressed > accordingly to a configured handler [hnd=AbstractFailureHandler > [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, > SYSTEM_CRITICAL_OPERATION_TIMEOUT]], failureCtx=FailureContext > [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker > [name=disco-event-worker, igniteInstanceName=null, finished=false, > heartbeatTs=1647956012230]]] > org.apache.ignite.IgniteException: GridWorker [name=disco-event-worker, > igniteInstanceName=null, finished=false, heartbeatTs=1647956012230] > at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method) > at java.base@11.0.9 > /java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) > at > org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.remove0(GridDhtAtomicCache.java:746) > >2022-03-22 13:33:51.701 WARN [tcp-disco-msg-worker-[5bc334df 0:0:0:0:0:0 > :0:1%lo:47500]-#2-#45] o.a.i.i.p.f.FailureProcessor - No deadlocked > threads detected. > ... > A thread dump where the blocked thread thread dump is: > Thread [name="disco-event-worker-#50", id=306, state=WAITING, blockCnt=1, > waitCnt=543567] > at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method) > at java.base@11.0.9 > /java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) > at > o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) > at > o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.remove0(GridDhtAtomicCache.java:746) > at > o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3397) > at > o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3380) > at > o.a.i.i.processors.cache.IgniteCacheProxyImpl.remove(IgniteCacheProxyImpl.java:1437) > at > o.a.i.i.processors.cache.GatewayProtectedCacheProxy.remove(GatewayProtectedCacheProxy.java:952) > at > es.decide.dwm.service.IgniteManager.removeProcessingRequest(IgniteManager.java:186) > at > es.decide.dwm.service.IgniteManager.lambda$registerIgniteEvents$1(IgniteManager.java:178) > at > es.decide.dwm.service.IgniteManager$$Lambda$8656/0x00080019d040.accept(Unknown > Source) > at java.base@11.0.9 > /java.util.ArrayList.forEach(ArrayList.java:1541) > at > es.decide.dwm.service.IgniteManager.lambda$registerIgniteEvents$3c60aaa7$1(IgniteManager.java:176) > at > es.decide.dwm.service.IgniteManager$$Lambda$3251/0x000802113c40.apply(Unknown > Source) > at > o.a.i.i.GridEventConsumeHandler$2.onEvent(GridEventConsumeHandler.java:192) > at >
Re: checkpointing timedout
Hi Surinder, > I increased it, but it started showing timeout errors These messages are INFO messages but not errors. Checkpoints can be triggered by several causes, some of them are below: 1) Checkpoint timeout. You specified 'checkpointFrequency' equal to 30 seconds, which means that checkpoint should be attempted at least each 30 seconds. 2) Too many dirty pages. 3) Creating a snapshot. 4) WAL segments rollover. ср, 23 мар. 2022 г. в 18:10, Surinder Mehra : > Hi, > We have a 4 node cluster with 64G RAM and 40G DISK per node attached for > /work and /walarchive each. WAL dir is 10G per node > > Below is our data region configuration and jvm_opts. We are getting > timeouts on checkpointing and WalArchive is getting filled up and no data > is moving to the /work directory. Checkpointing error is mentioned below > > could you suggest what's wrong with these configs(Note : with > walarchveSize as 16G and checkpointingBUffSize as 4G, it was writing but > very slow and throttling rate was 35% so I increased it, but it started > showing timeout errors) > > JVM_OPTS: -XX:MaxDirectMemorySize=2g -Xms20g -Xmx25g > -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC > -XX:+DisableExplicitGC > > Ignite config: > > > class="org.apache.ignite.configuration.DataStorageConfiguration"> > > > > > > > > class="org.apache.ignite.configuration.DataRegionConfiguration"> > > > > > > > value="/ignite/walarchive"/> > > > > > > Checkpointing logs: > > [14:56:55,209][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint > started [checkpointId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653, > startPtr=WALPointer [idx=930, fileOff=527943956, len=51411], > checkpointBeforeLockTime=9ms, checkpointLockWait=0ms, > checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=9ms, > walCpRecordFsyncDuration=7ms, writeCheckpointEntryDuration=3ms, > splitAndSortCpPagesDuration=45ms, pages=35542, reason='timeout'] > [14:56:55,921][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint > finished [cpId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653, pages=35542, > markPos=WALPointer [idx=930, fileOff=527943956, len=51411], > walSegmentsCovered=[], markDuration=64ms, pagesWrite=108ms, fsync=604ms, > total=785ms] >
Re: What this error says, and when it happens (Ignite 2.9.0, Java 8, Ubuntu 14.04)? SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT
Hi Aliaksandr, Were there any other errors or warnings in the ignite log? Maybe, there were messages about 'too long JVM pauses'? пт, 1 апр. 2022 г. в 17:46, Aliaksandr Kazlou : > Hi, recently in prod we started to observe the following error below. > What is it? And what could be the cause? Are there any recommendations > where to look/how to fix? > > > 2022-04-01 12:03:08.535 WARN --- [server.app%] ROOT > : Possible failure suppressed accordingly to a configured > handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class > o.a.i.IgniteException: GridWorker [name=sys-stripe-8, igniteInstanceName= > server.app, finished=false, heartbeatTs=1648814504489]]] > org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-8, > igniteInstanceName=server.app, finished=false, heartbeatTs=1648814504489] > at > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806) > at > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801) > at > org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234) > at > org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$1(ServerImpl.java:2970) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8057) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3086) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7995) > at > org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58) > 2022-04-01 12:03:08.541 WARN --- [er.app%] > o.a.i.i.p.cache.CacheDiagnosticManager : Page locks dump: > Thread=[name=exchange-worker-#67%server.app%, id=115], state=TIMED_WAITING >
Re: Return result stream from ignite compute
Hi Surinder,
Your tasks and nested classes should be in the class path of the
server node. You can deploy them manually or automatically by means of
peer-class loading [1].
1. https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading
пт, 1 апр. 2022 г. в 15:12, Surinder Mehra :
> Hi, I am trying to return the result stream from ignite compute task. When
> compute task has a map() on stream it fails with below error . Can someone
> please explain.
>
> "Exception in thread "main" class
> org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
> object [typeName=java.util.stream.ReferencePipeline$3]"
>
> Setup: One default ignite node
> Compute task
> client node to submit compute task and collect the stream results.
>
> public class StreamTask implements IgniteCallable> {
> @Override
> public Stream call() throws Exception {
>
> Function integerDepartmentFunction = i -> new
> MyFunction().apply(i);
> return IntStream.of(1, 2, 3).boxed().map(integerDepartmentFunction);
> }
> }
>
> class MyFunction implements Function {
> @Override
> public Department apply(Integer integer) {
> return new Department(integer, "sdf"+ integer);
> }
> }
>
>
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setClientMode(true);
> try (Ignite ignite = Ignition.start(cfg)) {
>
> ClusterGroup serversGrp = ignite.cluster().forServers();
>
> Stream stream = ignite.compute(serversGrp).call(new
> StreamTask());
> System.out.println("Stream : "+ stream.collect(Collectors.toList()));
>
> }
>
> Error:
>
> Exception in thread "main" class
> org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object
> [typeName=java.util.stream.ReferencePipeline$3]
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:971)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1769)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1721)
> at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:319)
> at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:304)
> at
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:101)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10540)
>
> Caused by: java.lang.ClassNotFoundException:
> training.ignite.compute.StreamTask$$Lambda$894/0x000800670840
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:398)
> at
> org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9064)
> at
> org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9002)
> at
> org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:376)
> at
> org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:693)
>
>
DML Statement error
Hi Igniters: First, I use custom primary key classes and BinaryObject to insert several pieces of data. And then, when I use the INSERT statement to insert data, the following error occurs: Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node. at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:340) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1918) ... 37 more Suppressed: class org.apache.ignite.binary.BinaryObjectException: Failed to get field because type ID of passed object differs from type ID this BinaryField belongs to [expected=[typeId=-181945856, typeName=com.inspur.edp.qdp.config.api.ketype.PubPartionKeys_1_7], actual=[typeId=1324872999, typeName=com.inspur.edp.qdp.config.keytype.PubPartionKeys_1_7], fieldId=-435253668, fieldName=TBDATA_DX01, fieldType=null] at org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302) at org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110) at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:223) at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:120) at org.apache.ignite.internal.processors.query.h2.opt.GridH2RowDescriptor.columnValue(GridH2RowDescriptor.java:235) at org.apache.ignite.internal.processors.query.h2.index.QueryIndexRowHandler.getKey(QueryIndexRowHandler.java:110) at org.apache.ignite.internal.processors.query.h2.index.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:73) at org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:68) at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.onUpdate(InlineIndexImpl.java:248) at org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndex(IndexProcessor.java:452) at org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndexes(IndexProcessor.java:295) at org.apache.ignite.internal.cache.query.index.IndexProcessor.store(IndexProcessor.java:142) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2549) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:422) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2666) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1742) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1717) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:441) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2327) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2553) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2016) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1833) Who knows why？Thanks！ Hty， 2022/04/02
