[
https://issues.apache.org/jira/browse/CLOUDSTACK-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123046#comment-15123046
]
Balaji Mani commented on CLOUDSTACK-9250:
-----------------------------------------
We have been getting Java Heap Space Out of Memory Error in CloudStack
Management Server frequently with CS release 4.5.2
We have providing load to CloudStack Management Server with below screnarios.
01. VM creation, VM Deletion, Volume Creation, Volume Deletion via Selenuim
automated script. The cloudstack management server runs for a day and later
mangment service gets down due to Java Heap Space Out of Memory Error.
02. External API calls to Cloudstack Management Server for VM creation, VM
Deletion, Volume Creation, Volume Deletion. The cloudstack management server
runs for few hours and later mangment service gets down due to Java Heap Space
Out of Memory Error.
03. API Calls via jar file for VM creation, VM Deletion, VM Rename, Volume
Creation, Volume Deletion. The cloudstack management server runs for few hours
and later mangment service gets down due to Java Heap Space Out of Memory Error.
We have enclosed hprof dump file that was generated during Java Heap Space Out
of Memory Error.
Also please see below log from cloudstack management server
2016-01-28 14:35:55,537 DEBUG [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-5333ae5d) Management server heartbeat takes too long
to finish. profiler: Done. Duration: 27789ms, profilerHeartbeatUpdate: Done.
Duration: 11214ms, profilerPeerScan: Done. Duration: 16575ms
2016-01-28 14:41:37,336 DEBUG [c.c.s.StatsCollector]
(StatsCollector-4:ctx-4148931f) AutoScaling Monitor is running...
2016-01-28 14:45:03,393 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-651ae6ea) HostStatsCollector is running...
2016-01-28 14:48:13,427 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(HostReservationReleaseChecker:ctx-177a3b51) Checking if any host reservation
can be released ...
2016-01-28 14:48:13,427 DEBUG [c.c.a.AlertManagerImpl]
(CapacityChecker:ctx-027f198c) Running Capacity Checker ...
2016-01-28 14:48:13,428 DEBUG [c.c.a.AlertManagerImpl]
(CapacityChecker:ctx-027f198c) recalculating system capacity
2016-01-28 14:48:13,428 DEBUG [c.c.a.AlertManagerImpl]
(CapacityChecker:ctx-027f198c) Executing cpu/ram capacity update
2016-01-28 14:48:16,003 INFO [c.c.h.HighAvailabilityManagerImpl]
(HA-4:ctx-003e359d) checking health of usage server
2016-01-28 14:51:38,249 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-bcb1b8b3) StorageCollector is running...
2016-01-28 14:55:07,978 ERROR [c.c.a.AlertManagerImpl]
(CapacityChecker:ctx-027f198c) Caught exception in recalculating capacity
java.lang.OutOfMemoryError: Java heap space
2016-01-28 14:55:14,423 WARN [c.c.s.VolumeApiServiceImpl]
(catalina-exec-17:ctx-00709386 ctx-ea1c8ed4 ctx-5bc68d39) Failed to expunge
volume:
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:851)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:1327)
at
com.cloud.storage.dao.VolumeDaoImpl.updateState(VolumeDaoImpl.java:485)
at
com.cloud.storage.dao.VolumeDaoImpl.updateState(VolumeDaoImpl.java:55)
at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy62.updateState(Unknown Source)
at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:126)
at
com.cloud.storage.VolumeApiServiceImpl.stateTransitTo(VolumeApiServiceImpl.java:1163)
at
com.cloud.storage.VolumeApiServiceImpl.deleteVolume(VolumeApiServiceImpl.java:1130)
at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy185.deleteVolume(Unknown Source)
at
org.apache.cloudstack.api.command.user.volume.DeleteVolumeCmd.execute(DeleteVolumeCmd.java:85)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:704)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:529)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:288)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:119)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:116)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:435)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at
org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:75)
at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at
com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:560)
at
com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:471)
at
com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:464)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:830)
... 60 more
2016-01-28 14:55:14,422 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-0a8d7999) Found the following agents behind on ping: [1]
2016-01-28 14:55:12,221 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgent-251:ctx-e7d4836a) Seq 1-1957939937999348093: Throwable caught
while executing command
java.lang.OutOfMemoryError: Java heap space
2016-01-28 14:55:10,748 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-79a685c7) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Storage automation results in java memory leaks.
> ------------------------------------------------
>
> Key: CLOUDSTACK-9250
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9250
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation
> Affects Versions: 4.5.2, 4.6.0
> Reporter: Balaji Mani
> Priority: Critical
>
> While automating the storage creation and deletion in selenium, after 1000
> storage creation and deletion, the CloudStack management server getting
> freezes due to JAVA Heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
