Hi Nuwan, Can we get a thread dump or attach a profiler while spinning? This seems like a different issue from the issues we have fixed so far.
On Mon, Sep 16, 2013 at 5:41 PM, Nuwan Silva <[email protected]> wrote: > I checked with the latest ELB (14th) which was downloaded from [1]. The > ELB spinning issue was still reproducible. > > Please Note, This is a setup with 2 Greg's fronted by a ELB. > To reproduce we just created some tenants and logged in with them a couple > of times. > > NOTE: refer jconsole attachment. > > top - 17:16:42 up 7:59, 16 users, load average: 3.38, 1.75, 1.00 > Tasks: 255 total, 2 running, 252 sleeping, 0 stopped, 1 zombie > %Cpu(s): 79.7 us, 1.5 sy, 0.0 ni, 18.6 id, 0.1 wa, 0.0 hi, 0.1 si, > 0.0 st > KiB Mem: 7871556 total, 7652276 used, 219280 free, 114340 buffers > KiB Swap: 8077308 total, 61824 used, 8015484 free, 2115372 cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 26703 nuwan 20 0 6372m 684m 15m S 291.7 8.9 8:48.20 > java > > 1961 nuwan 20 0 935m 155m 48m S 7.9 2.0 4:49.46 > chromium-browse > > 1937 nuwan 20 0 1520m 88m 21m S 6.3 1.2 12:14.97 > compiz > > 1173 root 20 0 536m 147m 116m S 5.6 1.9 14:56.26 Xorg > > process 26703 is ELB: > > nuwan@nuwan-ThinkPad-T530:~/ > projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/bin$ ps -ef | grep 26703 > nuwan 26703 26679 5 15:26 pts/2 00:05:50 > /home/nuwan/installation/java/jdk1.7.0_07/bin/java -Xbootclasspath/a: -d64 > -server -Xms1500m -Xmx3000m -XX:PermSize=256m -XX:MaxPermSize=512m > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:-UseGCOverheadLimit > -XX:+CMSClassUnloadingEnabled -XX:+OptimizeStringConcat > -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 26679;nohup > ./wso2server.sh & > -XX:HeapDumpPath=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/repository/logs/heap-dump.hprof > -XX:ErrorFile=repository/logs/hs_err_pid.log -XX:OnError=nohup > ./wso2server.sh & -Dcom.sun.management.jmxremote -classpath > /home/nuwan/installation/java/jdk1.7.0_07/lib/tools.jar:/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/bin/org.wso2.carbon.bootstrap-4.2.0.jar:/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/bin/tcpmon-1.0.jar:/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/bin/tomcat-juli-7.0.34.jar:/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/lib/commons-lang-2.6.0.wso2v1.jar > -Dwso2.loadbalancer=true > -Djava.endorsed.dirs=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/lib/endorsed:/home/nuwan/installation/java/jdk1.7.0_07/jre/lib/endorsed:/home/nuwan/installation/java/jdk1.7.0_07/lib/endorsed > -Djava.io.tmpdir=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/tmp > -Dcatalina.base=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/lib/tomcat > -Dwso2.server.standalone=true -Dcarbon.registry.root=/ > -Djava.command=/home/nuwan/installation/java/jdk1.7.0_07/bin/java > -Dcarbon.home=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0 > -Dloadbalancer.conf=file:repository/conf/loadbalancer.conf > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Dcarbon.config.dir.path=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/repository/conf > -Djava.util.logging.config.file=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/repository/conf/etc/logging-bridge.properties > -Dcomponents.repo=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/repository/components/plugins > -Dconf.location=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/repository/conf > -Dcom.atomikos.icatch.file=/home/nuwan/projects/carbon_4.2.0/cluster/elb/wso2elb-2.1.0/lib/transactions.properties > -Dcom.atomikos.icatch.hide_init_file_path=true > -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true > -Dcom.sun.jndi.ldap.connect.pool.authentication=simple > -Dcom.sun.jndi.ldap.connect.pool.timeout=3000 > -Dorg.terracotta.quartz.skipUpdateCheck=true > -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 > -DreverseProxyMode=true org.wso2.carbon.bootstrap.Bootstrap > nuwan 30751 6849 0 17:15 pts/10 00:00:00 grep --color=auto 26703 > > > Herewith i have attached my configuration files. If this is due to some > configuration issue please check and let me know. > > > [1] http://builder3.us1.wso2.org/builds/chunk-02 > /N_14-09-13/ELB/wso2elb-2.1.0_N20130914.zip > > Thanks, > NuwanS. > > > On Mon, Sep 16, 2013 at 9:25 AM, Nuwan Silva <[email protected]> wrote: > >> Hi Ravi, >> >> As said in the above mail i setup a Greg Cluster with 2 nodes and the >> latest ELB we got from [1]. >> Logged in as admin and created a a tenant and after that only worked with >> tenant. also logged in with a chrome browser and then this issue first >> occurred. >> >> restarted. then only worked with firefox browser but in around 10-15 >> minutes this issue was observed again. >> >> hope this helps since in my setup i get this often. >> >> [1] http://builder3.us1.wso2.org/builds/chunk-02/N_12-09-13/ELB/ >> >> Thanks, >> NuwanS. >> >> >> On Sun, Sep 15, 2013 at 2:14 PM, Ravi Undupitiya <[email protected]> wrote: >> >>> Hi Nuwan, >>> >>> I checked to see if the pack contained the spinning related fixes, and >>> they seem to be there. This might be due to another reason. >>> >>> It would be helpful if you can give the steps to reproduce this. >>> >>> >>> >>> On Sat, Sep 14, 2013 at 11:07 AM, Nuwan Silva <[email protected]> wrote: >>> >>>> Hi All, >>>> >>>> We setup a GREG cluster with the 2 nodes and given ELB 2.1.0 [1]. After >>>> working with it for sometime (5-10 minutes) ELB spinning issue appears >>>> please refer the attache screen-shot. >>>> >>>> console logs: >>>> [2013-09-14 10:48:29,636] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-1 >>>> [2013-09-14 10:54:26,351] WARN - SourceHandler I/O error (Probably the >>>> connection was closed by the remote party):Broken pipe >>>> [2013-09-14 10:54:29,755] WARN - SourceHandler Connection time out >>>> while writing the response: http-incoming-49 >>>> [2013-09-14 10:54:47,081] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-47 >>>> [2013-09-14 10:55:04,787] WARN - SourceHandler Connection time out >>>> while writing the response: http-incoming-43 >>>> [2013-09-14 10:55:27,036] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-46 >>>> [2013-09-14 10:56:27,102] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-55 >>>> [2013-09-14 10:56:28,099] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-57 >>>> [2013-09-14 10:57:45,575] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-59 >>>> [2013-09-14 10:57:48,592] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-61 >>>> [2013-09-14 10:58:48,645] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-60 >>>> [2013-09-14 10:59:48,706] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-63 >>>> [2013-09-14 10:59:59,677] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-65 >>>> [2013-09-14 11:01:04,820] WARN - SourceHandler Connection time out >>>> after request is read: http-incoming-67 >>>> >>>> >>>> [1] http://builder3.us1.wso2.org/builds/chunk-02/N_12-09-13/ELB/ >>>> >>>> >>>> On Fri, Sep 13, 2013 at 5:45 PM, Isuru Perera <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I noticed some of the >>>>> issues<https://wso2.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+10210+AND+labels+%3D+passthru-transport>were >>>>> marked as fixed today. Thanks for looking in to those. >>>>> >>>>> There are few in open state still. For example, >>>>> https://wso2.org/jira/browse/LB-88. This is probably the same issue >>>>> with spinning? >>>>> >>>>> Please check the NPEs as well. >>>>> >>>>> https://wso2.org/jira/browse/LB-85 >>>>> https://wso2.org/jira/browse/LB-105 >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> On Sun, Sep 8, 2013 at 8:34 PM, Isuru Perera <[email protected]> wrote: >>>>> >>>>>> Hi Ravi, >>>>>> >>>>>> Could you please go through the issues here: >>>>>> https://wso2.org/jira/browse/LB-106?jql=project%20%3D%2010210%20AND%20labels%20%3D%20passthru-transport >>>>>> >>>>>> I created a label for passthru transport related issues. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> On Thu, Sep 5, 2013 at 5:30 PM, Ravi Undupitiya <[email protected]>wrote: >>>>>> >>>>>>> LB-97 <https://wso2.org/jira/browse/LB-97> - this has been fixed. >>>>>>> >>>>>>> LB-106 <https://wso2.org/jira/browse/LB-106>, >>>>>>> LB-75<https://wso2.org/jira/browse/LB-75> >>>>>>> , LB-90 <https://wso2.org/jira/browse/LB-90> and >>>>>>> LB-105<https://wso2.org/jira/browse/LB-105> - should >>>>>>> be due to the spinning issue, i'm re-testing these scenarios with the >>>>>>> spinning fix to see if they reoccur on RC8 pack. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 5, 2013 at 12:05 PM, Isuru Perera <[email protected]>wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I have started working on fixing issues on ELB. >>>>>>>> >>>>>>>> Following are some synapse related issues that needs to be fixed. >>>>>>>> >>>>>>>> ESB Team, please check these issues and let me know whether those >>>>>>>> can be fixed. >>>>>>>> >>>>>>>> https://wso2.org/jira/browse/LB-74 >>>>>>>> https://wso2.org/jira/browse/LB-97 >>>>>>>> https://wso2.org/jira/browse/LB-106 >>>>>>>> https://wso2.org/jira/browse/LB-75 >>>>>>>> https://wso2.org/jira/browse/LB-90 >>>>>>>> https://wso2.org/jira/browse/LB-105 >>>>>>>> >>>>>>>> I will check on other issues and update this thread. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> -- >>>>>>>> Isuru Perera >>>>>>>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>>>>>>> Lean . Enterprise . Middleware >>>>>>>> >>>>>>>> about.me/chrishantha >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Ravi Undupitiya* >>>>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>>>> * >>>>>>> * >>>>>>> *E-mail: [email protected] >>>>>>> **M: **+94 772 930 712* >>>>>>> * >>>>>>> *Lean . Enterprise . Middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Isuru Perera >>>>>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>>>>> Lean . Enterprise . Middleware >>>>>> >>>>>> about.me/chrishantha >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Isuru Perera >>>>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>>>> Lean . Enterprise . Middleware >>>>> >>>>> about.me/chrishantha >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> * >>>> * >>>> *Nuwan Silva >>>> * >>>> *Senior Software Engineer - QA* >>>> Mobile: +94779804543 >>>> >>>> WSO2 Inc. >>>> lean . enterprise . middlewear. >>>> http://www.wso2.com >>>> >>> >>> >>> >>> -- >>> *Ravi Undupitiya* >>> Software Engineer; WSO2 Inc.; http://wso2.com >>> * >>> * >>> *E-mail: [email protected] >>> **M: **+94 772 930 712* >>> * >>> *Lean . Enterprise . Middleware >>> >> >> >> >> -- >> * >> * >> *Nuwan Silva >> * >> *Senior Software Engineer - QA* >> Mobile: +94779804543 >> >> WSO2 Inc. >> lean . enterprise . middlewear. >> http://www.wso2.com >> > > > > -- > * > * > *Nuwan Silva > * > *Senior Software Engineer - QA* > Mobile: +94779804543 > > WSO2 Inc. > lean . enterprise . middlewear. > http://www.wso2.com > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Kasun Indrasiri Software Architect WSO2, Inc.; http://wso2.com lean.enterprise.middleware cell: +94 71 536 4128 Blog : http://kasunpanorama.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
