Hi Ilya, I supposed following: GridDeploymentClassLoader usage for loading message class on receiver side is mandatory condition for reproduction. But for now it is only hypothesis.
In my test if all nodes have message payload class in its classpath there are no problems with messaging under high load (around 10^6 of messages in 32 threads). Same is true if application class loader is used in for loading message class (no WAR files & no Jetty). Could you please check class loaders used at sender(s) sides and on receiver side. Is GridDeploymentClassLoader used for loading message payload class at receiver? Sincerely, Dmitry Pavlov вт, 16 мая 2017 г. в 12:17, Ilya <iyakupo...@gmail.com>: > Hi Dmitry, > > Unfortunately, I've did not yet manage to reproduce this issue outside of > our project. > > What do you mean by "GridDeploymentClassLoader is used for loading class > on server"? How can server classloaders be configured? How does a remote > node choose, in which classloader to deploy the received class? > > My test configuration is as follows: > > - Web application that has a single cache and has two IgniteMessaging > local listeners; > - It's run from JUnit test code under Jetty; > - The three instances form a cluster; > - One client sends messages to one of the topics (to the random node) > using IgniteMessaging. > > All of these works on a single JVM. I suggest that Jetty servers might use > dedicated classloaders for each web-app. I've tried to launch client from > both test code and another web-app under Jetty, but that did not change > anything. > > In fact, the failing test on our production application is launched in the > same manner: two exploded web-apps (client and server), three instances of > server app and all of these is run under Jettys in a single VM... > > On Sun, May 14, 2017 at 2:22 PM, Dmitry Pavlov [via Apache Ignite Users] > <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=12879&i=0>> wrote: > >> Hi, Ilya, >> >> I've tried to reproduce deployment problem in standalone project >> involving Ingnite.start() in several WAR files. But this test is still >> passing. >> >> It is still possible deployment problem can be reprdoced only >> when GridDeploymentClassLoader is used for loading class on server, and >> several different Web App class loaders are enabled on clients. >> >> Do you have standalone reproducer you can share? >> >> Best Regards, >> Dmitry Pavlov >> >> >> >> пт, 12 мая 2017 г. в 15:58, Ilya <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=12831&i=0>>: >> > Hi all! >>> >>> The question was originally asked (but not answered) on SO: >>> >>> http://stackoverflow.com/questions/43803402/how-does-peer-classloading-work-in-apache-ignite >>> >>> In short, we have "Failed to deploy user message" exceptions under high >>> load >>> in our project. >>> >>> Here is an overview of our architecture: >>> - Distributed cache on three nodes, all nodes run on a single workstation >>> (in this test); >>> - Workers on each node; >>> - Messaging between workers is done using IgniteMessaging (topic has the >>> type of String and I've tried both byte[] and ByteBuffer as a message >>> class); >>> - Client connects to the cluster and triggers some business logic, that >>> causes cross-node messaging, scan queries and MR jobs (using >>> IgniteCompute::broadcast). All of these may performed concurrently. >>> >>> I've tried both SHARED and CONTINUOUS deployment mode, but the result >>> remains the same. >>> >>> I've noticed lots of similar messages in the logs: >>> /2017-05-05 13:31:28 INFO org.apache.ignite.logger.java.JavaLogger info >>> Removed undeployed class: GridDeployment [ts=1493980288578, >>> depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@38815daa, >>> clsLdrId=36c3828db51-0d65e7d5-77bf-444d-9b8b-d18bde94ad13, userVer=0, >>> loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, >>> undeployed=true, usage=0] >>> ... >>> 2017-05-05 13:31:29 INFO org.apache.ignite.logger.java.JavaLogger info >>> Removed undeployed class: GridDeployment [ts=1493980289125, >>> depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@355f6680, >>> clsLdrId=1dd3828db51-1b20df7a-a98d-45a3-8ab6-e5d229945830, userVer=0, >>> loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, >>> undeployed=true, usage=0] >>> .../ >>> >>> This happens when I use ByteBuffer as message type. In case of byte[], >>> class >>> B[ is being constantly re-deployed. >>> >>> ScanQuery predicate and IgniteCompute caller are also being constantly >>> re-deployed. >>> If we disable ScanQueries and IgniteCompute broadcasts - all is fine, >>> there >>> are no re-deployments. >>> >>> For the further testing I've disabled MRs and kept ScanQueries. I've also >>> added some debug output to a fresh snapshot of Ignite 2.1.0. Messages >>> "Class >>> locally deployed: <my ScanQuery predicate>" usually come from the >>> following >>> call stack: >>> >>> /org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.recordDeploy(GridDeploymentLocalStore.java:404) >>> at >>> >>> org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.deploy(GridDeploymentLocalStore.java:333) >>> at >>> >>> org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.getDeployment(GridDeploymentLocalStore.java:201) >>> at >>> >>> org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getLocalDeployment(GridDeploymentManager.java:383) >>> at >>> >>> org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getDeployment(GridDeploymentManager.java:345) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.injectResources(GridCacheQueryManager.java:918) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.scanIterator(GridCacheQueryManager.java:826) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.executeQuery(GridCacheQueryManager.java:611) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.queryResult(GridCacheQueryManager.java:1593) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1164) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:231) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109) >>> at >>> >>> org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107) >>> at >>> >>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863) >>> at >>> >>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386) >>> at >>> >>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308) >>> at >>> >>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100) >>> at >>> >>> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253) >>> at >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257) >>> at >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885) >>> at >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:114) >>> at >>> >>> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:802) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)/ >>> >>> Messages like "removed undeployed class" usually come from the >>> IgniteMessaging's call stack. >>> >>> I've analyzed the Ignite kernel a bit, and got a suspicion that undeploy >>> is >>> being triggered for all classes in a classloader, when at least one class >>> that resides in that classloader was re-deployed in some other loader. >>> >>> It happens inside >>> org.apache.ignite.spi.deployment.local.LocalDeploymentSpi#register >>> >>> At first, we get a "Map of new resources added for registered class >>> loader" using LocalDeploymentSpi#addResource. >>> Then we "Remove resources for all class loaders except {@code >>> ignoreClsLdr}." using LocalDeploymentSpi#removeResources. Inside this >>> method, it looks like we add all loaders that contain the old version of >>> the >>> new resource to a "doomed" collection. >>> Finally, we iterate this collection and call onClassLoaderReleased >>> for >>> each element. The latter action actually causes all the classes to be >>> undeployed (finally causing the "Removed undeployed class" messages). >>> >>> I don't understand this concept. Why are there multiple classloaders? >>> Why do >>> we undeploy the whole classloader in such cases? >>> >>> I'd be grateful, if someone could explain, how does peer classloading >>> work >>> in Ignite "under the hood". >>> >>> P. S. I'm looking at the sources of a fresh snapshot of Ignite 2.1.0, but >>> the behavior is the same with the standard Ignite 1.9.0. >>> >>> P. P. S. Unfortunately, I've did not manage to reproduce this issue >>> outside >>> of our project yet. >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661.html >>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12831.html >> To unsubscribe from Understanding the mechanics of peer class loading, click >> here. >> NAML >> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Understanding the mechanics of peer > class loading > <http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12879.html> > Sent from the Apache Ignite Users mailing list archive > <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >
