On Mon, May 19, 2014 at 10:46 AM, Afkham Azeez <az...@wso2.com> wrote:
> Ah.. since the ELB is also part of the cluster, cache is getting > replicated to that as well. I'll take a look and see how this can be fixed. > Providing all relevant information would have saved a lot of time. All this > while the discussion was revolving around the issue showing up on the IS > node which was misleading. > - Setup details : Operating System : Linux 3.11.0-17-generic x86_64 GNU/Linux Java Version/Java VM : java version "1.7.0_03"/Java HotSpot(TM) 64-Bit Server VM (build 22.1-b02, mixed mode) Packs : WSO2 IS 5.0.0 beta 18th May DBMS/DBMS Driver :Mysql (registry mounted) Browser : Chrome Version 33.0.1750.117 Setup : Cluster setup (Two IS manager nodes fronted by WSO2 ELB 2.1.1) User store : Openldap - Used IS documentation ( https://docs.wso2.org/display/CLUSTER420/Clustering+Identity+Server) to setup the IS cluster - IS side error is in https://wso2.org/jira/browse/LB-155 - Error in ELB TID: [0] [ELB] [2014-05-18 22:27:33,269] ERROR {com.hazelcast.map.operation.MergeOperation} - [127.0.0.1]:4100 [wso2.is1.domain] java.lang.ClassNotFoundException: org.wso2.carbon.identity.provisioning.cache.ProvisioningConnectorCacheKey {com.hazelcast.map.operation.MergeOperation} com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: org.wso2.carbon.identity.provisioning.cache.ProvisioningConnectorCacheKey at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:190) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:192) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:154) at com.hazelcast.map.MapService.toObject(MapService.java:644) at com.hazelcast.map.operation.MergeOperation.run(MergeOperation.java:47) at com.hazelcast.spi.impl.OperationServiceImpl.doRunOperation(OperationServiceImpl.java:222) at com.hazelcast.spi.impl.OperationServiceImpl.access$800(OperationServiceImpl.java:55) at com.hazelcast.spi.impl.OperationServiceImpl$LocalOperationProcessor.run(OperationServiceImpl.java:655) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) at com.hazelcast.spi.impl.OperationServiceImpl$OperationThread.run(OperationServiceImpl.java:731) Caused by: java.lang.ClassNotFoundException: org.wso2.carbon.identity.provisioning.cache.ProvisioningConnectorCacheKey at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at org.eclipse.core.runtime.internal.adaptor.ContextFinder.loadClass(ContextFinder.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:83) at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:77) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347) at java.io.ObjectInputStream.readUnshared(ObjectInputStream.java:459) at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:187) ... 12 more - I'll attach config files if necessary. Thanks, Pavithra > > Azeez > > > On Mon, May 19, 2014 at 10:35 AM, Prabath Siriwardena <prab...@wso2.com>wrote: > >> Any solution for this...? >> >> >> On Mon, May 19, 2014 at 10:34 AM, Pavithra Madurangi >> <pavit...@wso2.com>wrote: >> >>> >>> >>> >>> On Mon, May 19, 2014 at 10:32 AM, Kishanthan Thangarajah < >>> kishant...@wso2.com> wrote: >>> >>>> Yes, we observed a similar error on ELB, when there was a cache cleanup >>>> on an IS node. Is the issue here is coming from one of the IS node or from >>>> ELB? >>>> >>> >>> Just checked the ELB logs and yes issue is there.. Error occurs in ELB >>> and one IS node. >>> >>>> >>>> >>>> On Mon, May 19, 2014 at 10:30 AM, Pradeep Fernando <prad...@wso2.com>wrote: >>>> >>>>> So i had a discussion with kishanthan. We saw the same issue sometime >>>>> back. The exception was in ELB side. >>>>> >>>>> >>>>> On Mon, May 19, 2014 at 10:22 AM, Pavithra Madurangi < >>>>> pavit...@wso2.com> wrote: >>>>> >>>>>> No didn't observe this in single node..Anyway most of the testing >>>>>> were done in cluster.. So I'm not sure whether this is reproducible in >>>>>> single node... >>>>>> >>>>>> Thanks, >>>>>> Pavithra >>>>>> >>>>>> >>>>>> On Mon, May 19, 2014 at 10:18 AM, Afkham Azeez <az...@wso2.com>wrote: >>>>>> >>>>>>> So this happens when there is more than one node only, or are you >>>>>>> seeing it with even a single node? >>>>>>> >>>>>>> >>>>>>> On Mon, May 19, 2014 at 10:08 AM, Pavithra Madurangi < >>>>>>> pavit...@wso2.com> wrote: >>>>>>> >>>>>>>> Yes this was reproduced in a homogeneous cluster. I'll again check >>>>>>>> this with today's pack as well. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Pavithra >>>>>>>> >>>>>>>> >>>>>>>> On Mon, May 19, 2014 at 7:10 AM, Prabath Siriwardena < >>>>>>>> prab...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Yes its a homogeneous cluster.. Asela is looking into the issue... >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> -Prabath >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, May 19, 2014 at 5:52 AM, Pradeep Fernando < >>>>>>>>> prad...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> I think this is a homogeneous cluster... if not, then it is the >>>>>>>>>> root cause. I remember sometime ago, chamath and kishanthan working >>>>>>>>>> on a >>>>>>>>>> similar kind of issue. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, May 19, 2014 at 1:09 AM, Afkham Azeez <az...@wso2.com>wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, May 19, 2014 at 1:05 AM, Prabath Siriwardena < >>>>>>>>>>> prab...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 19, 2014 at 12:59 AM, Afkham Azeez >>>>>>>>>>>> <az...@wso2.com>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> That could be one reason, and another way this can happen is >>>>>>>>>>>>> this cache entry getting replicated on to another node that >>>>>>>>>>>>> cannot see this >>>>>>>>>>>>> class, and the first time it tries to access the entry is when it >>>>>>>>>>>>> tries to >>>>>>>>>>>>> do a cleanup, and at that point, tries to load the class, and >>>>>>>>>>>>> then this >>>>>>>>>>>>> failure happens. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If this is the case.. do we have a solution for it..? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> All nodes in a single cluster should be homogenous; which means, >>>>>>>>>>> all nodes should see the same bundles/classes. Then this error will >>>>>>>>>>> never >>>>>>>>>>> happen. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, May 19, 2014 at 12:55 AM, Pradeep Fernando < >>>>>>>>>>>>> prad...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> The class, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> org.wso2.carbon.identity.provisioning.cache.ProvisioningEntityCacheKey >>>>>>>>>>>>>> >>>>>>>>>>>>>> properly exported ? i know its a dumb question. just >>>>>>>>>>>>>> checking... :) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, May 19, 2014 at 12:43 AM, Prabath Siriwardena < >>>>>>>>>>>>>> prab...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Appreciate a lot any clue on this...? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://wso2.org/jira/browse/IDENTITY-2454 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Error occurred while running CacheCleanupTask >>>>>>>>>>>>>>> com.hazelcast.nio.serialization.HazelcastSerializationException: >>>>>>>>>>>>>>> java.lang.ClassNotFoundException: >>>>>>>>>>>>>>> org.wso2.carbon.identity.provisioning.cache.ProvisioningEntityCacheKey >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:190) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>>>> Prabath >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>>>> Prabath >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Twitter : @prabath >>>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/in/prabathsiriwardena >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mobile : +94 71 809 6732 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://blog.facilelogin.com >>>>>>>>>>>>>>> http://blog.api-security.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> *Pradeep Fernando* >>>>>>>>>>>>>> Associate Technical Lead;WSO2 Inc.; http://wso2.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> blog: http://pradeepfernando.blogspot.com >>>>>>>>>>>>>> m: +94776603662 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Afkham Azeez* >>>>>>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>>>>>>>>> * <http://www.apache.org/>* >>>>>>>>>>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>>>>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919> blog: * >>>>>>>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>>>>>>>>>> *twitter: >>>>>>>>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>>>>>>>>>>> * linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>>>>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>>>>>>>>>> >>>>>>>>>>>>> *Lean . Enterprise . Middleware* >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>> Prabath >>>>>>>>>>>> >>>>>>>>>>>> Twitter : @prabath >>>>>>>>>>>> LinkedIn : http://www.linkedin.com/in/prabathsiriwardena >>>>>>>>>>>> >>>>>>>>>>>> Mobile : +94 71 809 6732 >>>>>>>>>>>> >>>>>>>>>>>> http://blog.facilelogin.com >>>>>>>>>>>> http://blog.api-security.org >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Afkham Azeez* >>>>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>>>>>>> * <http://www.apache.org/>* >>>>>>>>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919> blog: * >>>>>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>>>>>>>> *twitter: >>>>>>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>>>>>>>>> * linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>>>>>>>> >>>>>>>>>>> *Lean . Enterprise . Middleware* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Pradeep Fernando* >>>>>>>>>> Associate Technical Lead;WSO2 Inc.; http://wso2.com >>>>>>>>>> >>>>>>>>>> blog: http://pradeepfernando.blogspot.com >>>>>>>>>> m: +94776603662 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> Dev@wso2.org >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks & Regards, >>>>>>>>> Prabath >>>>>>>>> >>>>>>>>> Twitter : @prabath >>>>>>>>> LinkedIn : http://www.linkedin.com/in/prabathsiriwardena >>>>>>>>> >>>>>>>>> Mobile : +94 71 809 6732 >>>>>>>>> >>>>>>>>> http://blog.facilelogin.com >>>>>>>>> http://blog.api-security.org >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Pavithra Madurangi* >>>>>>>> Associate Technical Lead - QA. >>>>>>>> WSO2 Inc.: http://wso2.com/ >>>>>>>> Mobile: +94777207357 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Afkham Azeez* >>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>>> * <http://www.apache.org/>* >>>>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919> blog: * >>>>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>>>> *twitter: >>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>>>>> * linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>>>> >>>>>>> *Lean . Enterprise . Middleware* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Pavithra Madurangi* >>>>>> Associate Technical Lead - QA. >>>>>> WSO2 Inc.: http://wso2.com/ >>>>>> Mobile: +94777207357 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Pradeep Fernando* >>>>> Associate Technical Lead;WSO2 Inc.; http://wso2.com >>>>> >>>>> blog: http://pradeepfernando.blogspot.com >>>>> m: +94776603662 >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Kishanthan Thangarajah* >>>> Senior Software Engineer, >>>> Platform Technologies Team, >>>> WSO2, Inc. >>>> lean.enterprise.middleware >>>> >>>> Mobile - +94773426635 >>>> Blog - *http://kishanthan.wordpress.com >>>> <http://kishanthan.wordpress.com>* >>>> Twitter - *http://twitter.com/kishanthan >>>> <http://twitter.com/kishanthan>* >>>> >>> >>> >>> >>> -- >>> *Pavithra Madurangi* >>> Associate Technical Lead - QA. >>> WSO2 Inc.: http://wso2.com/ >>> Mobile: +94777207357 >>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Thanks & Regards, >> Prabath >> >> Twitter : @prabath >> LinkedIn : http://www.linkedin.com/in/prabathsiriwardena >> >> Mobile : +94 71 809 6732 >> >> http://blog.facilelogin.com >> http://blog.api-security.org >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Afkham Azeez* > Director of Architecture; WSO2, Inc.; http://wso2.com > Member; Apache Software Foundation; http://www.apache.org/ > * <http://www.apache.org/>* > *email: **az...@wso2.com* <az...@wso2.com> > * cell: +94 77 3320919 <%2B94%2077%203320919> blog: * > *http://blog.afkham.org* <http://blog.afkham.org> > *twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> > * linked-in: **http://lk.linkedin.com/in/afkhamazeez > <http://lk.linkedin.com/in/afkhamazeez>* > > *Lean . Enterprise . Middleware* > -- *Pavithra Madurangi* Associate Technical Lead - QA. WSO2 Inc.: http://wso2.com/ Mobile: +94777207357 -
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
