Hello! I have got a problem.
I have 3 ignite (1.7 ver) nodes in the test cluster. On one of them placed ignite server with NodeFilter class that is locally available but I can’t deploy service “km.service” with provided NodeFilter into the cluster (when number of ignite service > 2) ClassNotFoundException is thrown public class NodeFilter implements IgnitePredicate<ClusterNode> { private final String role; public NodeFilter(String role) { this.role = role; } @Override public boolean apply(ClusterNode clusterNode) { Set<String> roles = clusterNode.attribute("roles"); return roles != null && roles.contains(role); } } Peer class loading enabled on all machines <property name="peerClassLoadingEnabled" value="true"/> Here is stack trace [14:27:39] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=5bee909d851-0da7494f-eb39-421f-b5e9-e3d031837679, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601], val=[op=CREATE, val=UserCacheObjectImpl [val=GridServiceDeployment [nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, cfg=LazyServiceConfiguration [srvcClsName=com.example.Service, svcCls=Service, nodeFilterCls=NodeFilter]], hasValBytes=true]], prevVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@216d2097], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], val=null, startVer=1481110057935, ver=GridCacheVersion [topVer=92589167, time=1481110059287, order=1481110057935, nodeOrder=22], hash=1562456177, extras=null, flags=0]]], prepared=0, locked=false, nodeId=cc42a26f-32ad-4c01-9114-efc3f89c308b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22]]], explicitLock=false, dhtVer=null, last=false, near=false, clientFirst=false, node=cc42a26f-32ad-4c01-9114-efc3f89c308b]], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, thread=<failed to find active thread 1>, mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601], val=[op=CREATE, val=UserCacheObjectImpl [val=GridServiceDeployment [nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, cfg=LazyServiceConfiguration [srvcClsName=com.example.Service, svcCls=Service, nodeFilterCls=NodeFilter]], hasValBytes=true]], prevVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@216d2097], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], val=null, startVer=1481110057935, ver=GridCacheVersion [topVer=92589167, time=1481110059287, order=1481110057935, nodeOrder=22], hash=1562456177, extras=null, flags=0]]], prepared=0, locked=false, nodeId=cc42a26f-32ad-4c01-9114-efc3f89c308b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22]]], explicitLock=false, dhtVer=null, last=false, near=false, clientFirst=false, node=cc42a26f-32ad-4c01-9114-efc3f89c308b]], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22], writeVer=null, implicit=true, loc=true, threadId=1, startTime=1481110059294, nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, startVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=true, plc=5, commitVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22], finalizing=NONE, preparing=false, invalidParts=null, state=ROLLED_BACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=22, minorTopVer=0], duration=30ms, onePhaseCommit=false], size=1]]], commit=false, mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601], val=[op=CREATE, val=UserCacheObjectImpl [val=GridServiceDeployment [nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, cfg=LazyServiceConfiguration [srvcClsName=com.example.Service, svcCls=Service, nodeFilterCls=NodeFilter]], hasValBytes=true]], prevVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@216d2097], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], val=null, startVer=1481110057935, ver=GridCacheVersion [topVer=92589167, time=1481110059287, order=1481110057935, nodeOrder=22], hash=1562456177, extras=null, flags=0]]], prepared=0, locked=false, nodeId=cc42a26f-32ad-4c01-9114-efc3f89c308b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22]]], explicitLock=false, dhtVer=null, last=false, near=false, clientFirst=false, node=cc42a26f-32ad-4c01-9114-efc3f89c308b]], trackable=true, finishOnePhaseCalled=false, innerFuts=[FinishFuture[node=cc42a26f-32ad-4c01-9114-efc3f89c308b, loc=false, done=true]], super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=o.a.i.i.util.lang.GridFunc$27@5500a61a, initFlag=1, lsnrCalls=0, done=false, cancelled=false, err=null, futs=[true]]]]class org.apache.ignite.IgniteCheckedException: Transaction has been already completed. at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:776) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:718) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:681) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:156) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:154) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:748) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:353) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:277) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:88) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:231) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1238) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:866) at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:106) at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:829) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 14:27:39.344 [main] ERROR o.a.i.i.p.c.d.d.c.GridDhtColocatedCache - <ignite-sys-cache> Failed to rollback transaction (cache may contain stale locks): GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601], val=[op=CREATE, val=UserCacheObjectImpl [val=GridServiceDeployment [nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, cfg=LazyServiceConfiguration [srvcClsName=com.example.Service, svcCls=Service, nodeFilterCls=NodeFilter]], hasValBytes=true]], prevVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@216d2097], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], val=null, startVer=1481110057935, ver=GridCacheVersion [topVer=92589167, time=1481110059287, order=1481110057935, nodeOrder=22], hash=1562456177, extras=null, flags=0]]], prepared=0, locked=false, nodeId=cc42a26f-32ad-4c01-9114-efc3f89c308b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22]]], explicitLock=false, dhtVer=null, last=false, near=false, clientFirst=false, node=cc42a26f-32ad-4c01-9114-efc3f89c308b]], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, thread=main, mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], cacheId=-2100569601], val=[op=CREATE, val=UserCacheObjectImpl [val=GridServiceDeployment [nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, cfg=LazyServiceConfiguration [srvcClsName=com.example.Service, svcCls=Service, nodeFilterCls=NodeFilter]], hasValBytes=true]], prevVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@216d2097], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=GridServiceDeploymentKey [name=km.service], hasValBytes=true], val=null, startVer=1481110057935, ver=GridCacheVersion [topVer=92589167, time=1481110059287, order=1481110057935, nodeOrder=22], hash=1562456177, extras=null, flags=0]]], prepared=0, locked=false, nodeId=cc42a26f-32ad-4c01-9114-efc3f89c308b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22]]], explicitLock=false, dhtVer=null, last=false, near=false, clientFirst=false, node=cc42a26f-32ad-4c01-9114-efc3f89c308b]], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22], writeVer=null, implicit=true, loc=true, threadId=1, startTime=1481110059294, nodeId=30a3b09f-8a4d-4f35-96ef-45a3dc9d7fc6, startVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=true, plc=5, commitVer=GridCacheVersion [topVer=92589167, time=1481110059286, order=1481110057933, nodeOrder=22], finalizing=NONE, preparing=false, invalidParts=null, state=ROLLED_BACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=22, minorTopVer=0], duration=40ms, onePhaseCommit=false], size=1]]] 14:27:17.326 [main] ERROR o.a.i.i.p.s.GridServiceProcessor - Failed to deploy service: km.service org.apache.ignite.IgniteCheckedException: Failed to unmarshal object with optimized marshaller at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1221) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:348) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:277) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:88) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:231) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1238) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:866) at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:106) at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:829) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1595) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1663) at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:298) at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal(BinaryMarshaller.java:109) at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:111) at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.unmarshal(CacheObjectBinaryProcessorImpl.java:766) at org.apache.ignite.internal.processors.cache.CacheObjectImpl.finishUnmarshal(CacheObjectImpl.java:109) at org.apache.ignite.internal.processors.cache.transactions.TxEntryValueHolder.unmarshal(TxEntryValueHolder.java:153) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxEntry.unmarshal(IgniteTxEntry.java:913) at org.apache.ignite.internal.processors.cache.GridCacheMessage.unmarshalTx(GridCacheMessage.java:408) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxPrepareRequest.finishUnmarshal(GridDistributedTxPrepareRequest.java:368) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1215) ... 11 common frames omitted Caused by: org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading): org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@29f10026 at org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:224) at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1592) ... 22 common frames omitted Caused by: java.lang.ClassNotFoundException: com.example.NodeFilter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader.findClass(GridCacheDeploymentManager.java:858) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader.loadClass(GridCacheDeploymentManager.java:815) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:8350) at org.apache.ignite.internal.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:185) at org.apache.ignite.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:266) at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:367) at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:491) at org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:579) at org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:841) With best regards Alisher Alimov alimovalis...@gmail.com