[
https://issues.apache.org/jira/browse/CLOUDSTACK-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592166#comment-13592166
]
Sowmya Krishnan commented on CLOUDSTACK-1496:
---------------------------------------------
Logs:
2013-03-04 18:08:23,838 WARN [cloud.resource.ResourceManagerImpl]
(AgentTaskPool-2:null) Unable to connect due to
com.cloud.utils.exception.CloudRuntimeException: Unable to connect 581
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:630)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy112.handleDirectConnectAgent(Unknown Source)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy122.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Caught:
com.mysql.jdbc.PreparedStatement@57a07056: SELECT vm_instance.id,
vm_instance.name, vm_instance.vnc_password, vm_instance.proxy_id,
vm_instance.proxy_assign_time, vm_instance.state,
vm_instance.private_ip_address, vm_instance.instance_name,
vm_instance.vm_template_id, vm_instance.guest_os_id, vm_instance.host_id,
vm_instance.last_host_id, vm_instance.pod_id, vm_instance.private_mac_address,
vm_instance.data_center_id, vm_instance.vm_type, vm_instance.ha_enabled,
vm_instance.limit_cpu_use, vm_instance.update_count, vm_instance.created,
vm_instance.removed, vm_instance.update_time, vm_instance.domain_id,
vm_instance.account_id, vm_instance.service_offering_id,
vm_instance.reservation_id, vm_instance.hypervisor_type, vm_instance.uuid,
vm_instance.disk_offering_id, domain_router.element_id,
domain_router.public_ip_address, domain_router.public_mac_address,
domain_router.public_netmask, domain_router.is_redundant_router,
domain_router.priority, domain_router.is_priority_bumpup,
domain_router.redundant_state, domain_router.stop_pending, domain_router.role,
domain_router.template_version, domain_router.scripts_version,
domain_router.vpc_id FROM domain_router INNER JOIN vm_instance ON
domain_router.id=vm_instance.id INNER JOIN router_network_ref ON
domain_router.id=router_network_ref.router_id WHERE
vm_instance.type='DomainRouter' AND vm_instance.host_id = 581 AND
vm_instance.removed IS NULL AND (router_network_ref.guest_type = 'Isolated' )
at
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:417)
at
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:350)
at
com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:902)
at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:879)
at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:892)
at
com.cloud.vm.dao.DomainRouterDaoImpl.listIsolatedByHostId(DomainRouterDaoImpl.java:217)
at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy156.listIsolatedByHostId(Unknown Source)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.processConnect(VirtualNetworkApplianceManagerImpl.java:3515)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
... 45 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
... 71 more
> List API Performance: listAccounts failing with OOME for high values of
> pagesize (>1000 )
> -----------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-1496
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1496
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: API
> Affects Versions: 4.1.0
> Environment: CentOS release 6.3
> Reporter: Sowmya Krishnan
> Assignee: Min Chen
> Fix For: 4.1.0
>
>
> listAccounts with higher values of pagesize is resulting in OOME and in
> general, showing low performance for smaller page size values too.
> Configuration for the performance set up is as follows:
> (Using simulator set up)
> (Advanced zone)
> Accounts: 2117
> Hosts: 1986
> Users: 2116
> Virtual machines: 3299
> Server configurations:
> Management Server :
> =================
> 8 proc Intel(R) Xeon(R) CPU E5620 @ 2.40GHz processor
> CentOS release 6.3 (Final)
> Database:
> ========
> 8 proc Intel(R) Xeon(R) CPU E5620 @ 2.40GHz processor
> Red Hat Enterprise Linux Server release 6.2 (Santiago)
> MySQL-server-5.5.21-1.linux2.6.x86_64 (InnoDB engine)
> Following are the results for listAccounts for various pagesize values:
> ======================================================
> (APIs are fired on 8080 as admin from a different client than the MS)
> pagesize=10 : 0m30.944s
> pagesize=50: 1m57.292s
> pagesize=100: 2m59.700s
> pagesize=500: 12m22.807s
> In comparison to the results before API Optimization, for pagesize=100, the
> response was received within 62 seconds before optimization whereas now, the
> result is 149 seconds
> One difference though, is that the performance run before API optimization
> was done on a DB server with much higher configuration (Quad core, 8
> processor). But still, we shouldn't be seeing OOME.
> Following is the extract of the result for listAccounts&listAll=true (# of
> accounts: 2117)
> ...
> <title>Error 500 Java heap space</title> </head> <body><h2>HTTP ERROR
> 500</h2> <p>Problem accessing /client/api. Reason: <pre> Java heap
> space</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError: Java heap
> space </pre> <hr /><i><small>Powered by Jetty://</small>
> Maven Settings:
> ============
> [root@centos63 ~]# echo $MAVEN_OPTS
> -Xmx5120m -XX:MaxPermSize=1024m -Xdebug
> -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
