Re: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
>>As a temporary solution you can stop relying on peer class loading for continuous queries and provide the code of remote filters to the classpath of server nodes. Yes.. I was thinking of a solution on similar lines. Thank you for the reply. -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
Thanks for the report! The issue here is that a remote filter for a continuous query is loaded using peer class loading, and other classes that this remote filter depends on can be lazily loaded during its work. Loading every dependency class involves going to the node where the originating class was loading from, and asking that node to send missing classes over the network. The issues begin when this node is not in the cluster anymore, and the continuous query wasn't undeployed yet. A server sends a request for a class to a node that is not available, but wasn't kicked out of the topology yet, since a failure detection timeout hasn't elapsed yet. It leads to a NoClasDefFound exception that you observe in the logs. The biggest issue here is that this exception triggers a failure handler that makes the whole node go down. I would expect that only one request would fail, but not the whole node. As a temporary solution you can stop relying on peer class loading for continuous queries and provide the code of remote filters to the classpath of server nodes. This way no lazy class loading will be performed over the network since they will all be available locally. Denis пт, 13 мар. 2020 г. в 20:39, VeenaMithare : > Raised this jira : > https://issues.apache.org/jira/browse/IGNITE-12784 > > Observed in 2.7.6. Unable to easily test in 2.8.0 because of other issues. > One of them being - > > http://apache-ignite-users.70518.x6.nabble.com/2-8-0-JDBC-Thin-Client-Unable-to-load-the-tables-via-DBeaver-td31681.html > Please note this happens > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Re: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
Raised this jira : https://issues.apache.org/jira/browse/IGNITE-12784 Observed in 2.7.6. Unable to easily test in 2.8.0 because of other issues. One of them being - http://apache-ignite-users.70518.x6.nabble.com/2-8-0-JDBC-Thin-Client-Unable-to-load-the-tables-via-DBeaver-td31681.html Please note this happens -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[jira] [Created] (IGNITE-12784) Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
Veena Mithare created IGNITE-12784: -- Summary: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error Key: IGNITE-12784 URL: https://issues.apache.org/jira/browse/IGNITE-12784 Project: Ignite Issue Type: Bug Affects Versions: 2.7.6 Reporter: Veena Mithare We have a 3 node server cluster ( Issue observed in 2.7.6, could not test in 2.8.0 because I am unable to bring up the dbeaver in 2.8.0 with securityplugin enabled )([http://apache-ignite-users.70518.x6.nabble.com/2-8-0-JDBC-Thin-Client-Unable-to-load-the-tables-via-DBeaver-td31681.html]) A 4th node joins as a client with a continuous query on a Table A( Transaction_mode = transactional ). Now If I bring the client down and issue an update to the Table A within failureDetectionTimeout 3 , I get the following error and *this error* *brings the server down:* "(err) Failed to notify listener: GridDhtTxPrepareFuture Error" === Basically the server , tries to update the record on the Table A, and tries to notify Client since it had registered a continuous query for Table A. But since the Client Node has been brought down, it undeploys the remotefilterfactory lambda. Hence the server is no longer able to complete the transaction . */This also brings the server down./ * How can I resolve this issue ? === Please find the complete stack trace for this error : 2020-03-13 17:13:40,145 [sys-stripe-19-#20] ERROR [] - Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NoClassDefFoundError: com/companyname/projectname/Module/helper/ContinuousQueryHelper]] java.lang.NoClassDefFoundError: com/companyname/projectname/Module/helper/ContinuousQueryHelper at com.companyname.projectname.Module.helper.ContinuousQueryHelper$ModuleTableRemoteFilterFactory$1.evaluate(ContinuousQueryHelper.java:289) ~[?:?] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.filter(CacheContinuousQueryHandler.java:833) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:422) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:426) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1584) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:741) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.localFinish(GridNearTxLocal.java:3646) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.doFinish(GridNearTxFinishFuture.java:475) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:425) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:3788) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:3782) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.GridCacheCompoundFuture.onDone(GridCacheCompoundFuture.java:56) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.onComplete(GridNearOptimisticTxPrepareFuture.java:307) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFut
Re: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
Hello! But why the node was down? Was it due to failure handler? Unhandled exception in critical thread? Anything else? Did you find any work-around? I have not heard about Continuous Query issues such as the one you are describing, and I suggest filing an IGNITE ticket with details. Regards, -- Ilya Kasnacheev чт, 12 мар. 2020 г. в 16:39, VeenaMithare : > Hi , > > Did anyone get a chance to look at this ? > Summary of the issue I am facing. : > > We have a 3 node server cluster > > A 4th node joins as a client with a continuous query on a Table A( > Transaction_mode = transactional ). > > Now If I bring the client down and issue an update to the Table A within > failureDetectionTimeout 3 , I get the following error and */this error > brings the server down/*: > > "(err) Failed to notify listener: GridDhtTxPrepareFuture Error" > === > Basically the server , tries to update the record on the Table A, and tries > to notify Client since it had registered a continuous query for Table A. > But since the Client Node has been brought down, it undeploys the > remotefilterfactory lambda. Hence the server is no longer able to complete > the > transaction . > > */This also brings the server down./ > * > regards, > Veena. > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Re: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
Hi , Did anyone get a chance to look at this ? Summary of the issue I am facing. : We have a 3 node server cluster A 4th node joins as a client with a continuous query on a Table A( Transaction_mode = transactional ). Now If I bring the client down and issue an update to the Table A within failureDetectionTimeout 3 , I get the following error and */this error brings the server down/*: "(err) Failed to notify listener: GridDhtTxPrepareFuture Error" === Basically the server , tries to update the record on the Table A, and tries to notify Client since it had registered a continuous query for Table A. But since the Client Node has been brought down, it undeploys the remotefilterfactory lambda. Hence the server is no longer able to complete the transaction . */This also brings the server down./ * regards, Veena. -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error
Hi Team, We have a 3 node server cluster A 4th node joins as a client with a continuous query on a Table A( Transaction_mode = transactional ). Now If I bring the client down and issue an update to the Table A within failureDetectionTimeout 3 , I get the following error and */this error brings the server down/*: "(err) Failed to notify listener: GridDhtTxPrepareFuture Error" === Basically the server , tries to update the record on the Table A, and tries to notify Client since it had registered a continuous query for Table A. But since the Client Node has been brought down, and it undeploys the remotefilterfactory lambda, and the server is no longer able to complete the transaction . */This also brings the server down./ * How can I resolve this issue ? === Please find the complete stack trace for this error : [12:14:12] (err) Failed to notify listener: GridDhtTxPrepareFuture [futId=0a69e79c071-93faf34d-a776-4166-9f3b-4b5a0f54b8f9, err=null, replied=1, mapped=1, req=GridNearTxPrepareRequest [futId=4250e79c071-51438f4f-c061-45f7-b34e-57c90f2055e9, miniId=1, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], subjId=da486d0b-36a1-43d4-b05b-47d126fd880e, taskNameHash=0, flags=[implicitSingle], super=GridDistributedTxPrepareRequest [threadId=382, concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion [topVer=195408427, order=1583928843624, nodeOrder=1], timeout=1000, reads=null, writes=[IgniteTxEntry [key=ABCKEY [idHash=1413504800, hash=-1419375634, VALUETYPE=somevaluetype, NAME=TEST4375234], cacheId=-1512899836, txKey=IgniteTxKey [key=ABCKEY [idHash=1413504800, hash=-1419375634, VALUETYPE=somevaluetype, NAME=TEST4375234], cacheId=-1512899836], val=[op=CREATE, val=ABC [idHash=108633195, hash=-965148880, ACTIVE=true, MODIFICATIONDATE=2020-02-03 18:29:03.501, VALUETYPE=null, SCHEMAREF=null, VALUE=DEV, MACHINENAME=null, COMMENT=null, NAME=null, APPLICATIONNAME=null, SCHEMANAME=null, KEYNAME=ENVIRONMENT, USERNAME=null, INTERNALVERSION=null, MODIFICATIONTYPE=null]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[o.a.i.i.processors.cache.CacheEntrySerializablePredicate@388c822f], filtersPassed=false, filtersSet=false, entry=GridDhtCacheEntry [rdrs=[], part=136, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=ABCKEY [idHash=1413504800, hash=-1419375634, VALUETYPE=somevaluetype, NAME=TEST4375234], val=null, ver=GridCacheVersion [topVer=195408427, order=1583928843625, nodeOrder=4], hash=-1419375634, extras=GridCacheObsoleteEntryExtras [obsoleteVer=GridCacheVersion [topVer=2147483647, order=0, nodeOrder=0]], flags=2]]], prepared=1, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=195408427, order=1583928843624, nodeOrder=1], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0, trackable=true, nearMiniId=1, last=true, retVal=false, ret=GridCacheReturn [v=null, cacheObj=null, success=true, invokeRes=false, loc=false, cacheId=0], lockKeys=[], forceKeysFut=null, locksReady=true, invoke=false, timeoutObj=PrepareTimeoutObject [timeout=1000], xid=GridCacheVersion [topVer=195408427, order=1583928843625, nodeOrder=4], innerFuts=[[node=da486d0b-36a1-43d4-b05b-47d126fd880e, loc=false, done=true]], super=GridCompoundFuture [rdc=o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$1@73415bf, initFlag=1, lsnrCalls=1, done=true, cancelled=false, err=null, futs=[true]]]java.lang.NoClassDefFoundError: com/companyname/abc/configstore/helper/ContinuousQueryHelper at com.companyname.abc.configstore.helper.ContinuousQueryHelper$ConfigStoreTableRemoteFilterFactory$1.evaluate(ContinuousQueryHelper.java:293) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.filter(CacheContinuousQueryHandler.java:833) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:422) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:426) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1584) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:741) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:796) at
