Hello! I have checked your reproducer, I were not able to run it since it has a lot of dependencies, but there's a lot of going there: You have a scheduled method, which is run periodically and which executes code on the thread pool which calls these entry processors.
I would recommend simplifying this use case until it works: such as, running entry processors from main/application thread, then adding thread pool, then adding schedule. This way you may identify the step which causes this issue. Maybe the issue is related to some class loading problem. Maybe some of classes' dependency cannot be peer loaded. Since these errors do not seem to contain stack trace, some debugging may be needed. Regards, -- Ilya Kasnacheev пн, 24 мая 2021 г. в 18:47, ihalilaltun <ibrahim.al...@segmentify.com>: > Hi, > > I've run more detailed tests during the weekend and i can surely tell that > problem is not related to the migrated data. With a new cluster setup and > with 0 data we can still get the error. > > what i have in my mind is this; with the new version there may be a new > configuration parameter that has to be set in order cacheentryprocessors to > be DEPLOYED in SHARED mode to all cluster nodes, but i cannot find such a > parameter. > > so at this point my problem becomes to this; is there a configuration > parameter that forces all cacheentroprocessors the be deployed on every > cluster node from client by force? > > following is the cluster and client configuration; > client-server-configs.zip > < > http://apache-ignite-users.70518.x6.nabble.com/file/t2515/client-server-configs.zip> > > > when the client nodes starts and run necessary jobs -containing > cacaheentryprocessors- first 1 or 2 cacheentryprocessors are deploed to > both > clusters after that new cacaheentryprocessors starts to get > classnotfoundexception and cluster nodes keeps giving following warnings; > > > [2021-05-24T15:36:00,010][WARN > ][sys-stripe-4-#5][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.ShiftPromotionCountersEntryProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870560005, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=627410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=627410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.ShiftPromotionCountersEntryProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:36:00,103][WARN > ][sys-stripe-2-#3][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870560100, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=a27410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=a27410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:36:00,180][WARN > ][sys-stripe-1-#2][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870560171, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=f27410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=f27410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:36:00,202][WARN > ][sys-stripe-1-#2][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870560191, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=137410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=137410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:36:00,308][WARN > ][sys-stripe-1-#2][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870560302, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=337410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=337410f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreResetProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:36:16,861][INFO ][grid-timeout-worker-#198][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=f4e082a1, uptime=00:04:00.184] > ^-- Cluster [hosts=8, CPUs=34, servers=2, clients=19, topVer=27, > minorTopVer=0] > ^-- Network [addrs=[10.240.242.191, 10.240.245.191, 127.0.0.1], > discoPort=47500, commPort=47100] > ^-- CPU [CPUs=4, curLoad=1.27%, avgLoad=13.55%, GC=0%] > ^-- Heap [used=1682MB, free=61.37%, comm=2023MB] > ^-- Off-heap memory [used=7427MB, free=71.43%, allocated=25800MB] > ^-- Page memory [pages=1879439] > ^-- sysMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, > allocRam=100MB, allocTotal=0MB] > ^-- default region [type=default, persistence=true, lazyAlloc=true, > ... initCfg=256MB, maxCfg=25600MB, usedRam=7427MB, freeRam=70.99%, > allocRam=25600MB, allocTotal=13476MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.89%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- volatileDsMemPlc region [type=user, persistence=false, > lazyAlloc=true, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=0MB] > ^-- Ignite persistence [used=13476MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=1, idle=6, qSize=0] > [2021-05-24T15:36:46,778][WARN > ][sys-stripe-28-#29][GridDeploymentPerVersionStore] Failed to load peer > class (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870606767, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=082510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=082510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:36:51,975][WARN > ][sys-stripe-21-#22][GridDeploymentPerVersionStore] Failed to load peer > class (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870611965, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=d25510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=d25510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:37:00,011][WARN > ][sys-stripe-4-#5][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.ShiftPromotionCountersEntryProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870620009, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=717510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=717510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.ShiftPromotionCountersEntryProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:37:03,124][WARN > ][sys-stripe-39-#40][GridDeploymentPerVersionStore] Failed to load peer > class (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870623120, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=b38510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=b38510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > [2021-05-24T15:37:10,256][WARN > ][sys-stripe-4-#5][GridDeploymentPerVersionStore] Failed to load peer class > (ignore if class got undeployed during preloading) > > [alias=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > dep=SharedDeployment [rmv=false, super=GridDeployment [ts=1621870630246, > depMode=SHARED, clsLdr=GridDeploymentClassLoader > [id=f4a510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, singleNode=false, > nodeLdrMap=HashMap > > {db22a85d-37a5-45c4-ae63-bdd535eaca44=75f020f9971-db22a85d-37a5-45c4-ae63-bdd535eaca44}, > p2pTimeout=5000, usrVer=0, depMode=SHARED, quiet=false], > clsLdrId=f4a510f9971-f4e082a1-4012-4720-bcbb-e438359221e1, userVer=0, > loc=false, > > sampleClsName=com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor, > pendingUndeploy=false, undeployed=false, usage=0]]] > > > what i need is to deploy all cacheentryprocessors to clusters either > gracefully or by force. > > > all cacheentryprocessors are called as in this service (i had to remove > some > part of it due to company regulations); > service.zip > <http://apache-ignite-users.70518.x6.nabble.com/file/t2515/service.zip> > > thanks > > > > ----- > İbrahim Halil Altun > Senior Software Engineer @ Segmentify > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
