Parth Jagirdar created CLOUDSTACK-6016: ------------------------------------------
Summary: MS: MS out of memory java.lang.OutOfMemoryError: PermGen space Key: CLOUDSTACK-6016 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6016 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Contrail, Management Server, XenServer Affects Versions: 4.2.1 Environment: Master with Contrail and Xen. Reporter: Parth Jagirdar Priority: Critical Fix For: 4.2.1 Detailed logs are attached .. +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+----------------+-----------+--------------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ | id | name | uuid | pool_type | port | data_center_id | pod_id | cluster_id | used_bytes | capacity_bytes | host_address | user_info | path | created | removed | update_time | status | storage_provider_name | scope | hypervisor | managed | capacity_iops | +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+----------------+-----------+--------------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ | 16 | primary | 15027b8a-bb16-3f01-b5af-e980a0060f25 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 4029850058752 | 11810778316800 | 10.223.110.232 | NULL | /export/home/parth/contrail/cloudstack_248_196/primary | 2014-01-31 20:53:34 | NULL | NULL | Up | DefaultPrimary | CLUSTER | NULL | 0 | NULL | +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+----------------+-----------+--------------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ 1 row in set (0.01 sec) mysql> select * from storage_pool_view; +----+--------------------------------------+---------+--------+--------------------------------------------------------+-------------------+----------------+---------------------+---------+----------------+---------------+---------+------------+-----------------------+------------+--------------------------------------+----------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+--------------------------------------+-----------+------+--------------------+------------------------+--------+----------+------------+----------------+ | id | uuid | name | status | path | pool_type | host_address | created | removed | capacity_bytes | capacity_iops | scope | hypervisor | storage_provider_name | cluster_id | cluster_uuid | cluster_name | cluster_type | data_center_id | data_center_uuid | data_center_name | data_center_type | pod_id | pod_uuid | pod_name | tag | disk_used_capacity | disk_reserved_capacity | job_id | job_uuid | job_status | job_account_id | +----+--------------------------------------+---------+--------+--------------------------------------------------------+-------------------+----------------+---------------------+---------+----------------+---------------+---------+------------+-----------------------+------------+--------------------------------------+----------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+--------------------------------------+-----------+------+--------------------+------------------------+--------+----------+------------+----------------+ | 16 | 15027b8a-bb16-3f01-b5af-e980a0060f25 | primary | Up | /export/home/parth/contrail/cloudstack_248_196/primary | NetworkFilesystem | 10.223.110.232 | 2014-01-31 20:53:34 | NULL | 11810778316800 | NULL | CLUSTER | NULL | DefaultPrimary | 1 | 878b925d-9373-4dc5-88f2-9a83d6d924b2 | demo-pod2-xen1 | CloudManaged | 1 | 8d2fe185-7f85-45e0-b523-cf318c66e799 | default | Advanced | 1 | fdd9ed6b-3618-4f9f-afe1-191eb290da22 | demo-pod2 | NULL | 0 | 0 | NULL | NULL | NULL | NULL | +----+--------------------------------------+---------+--------+--------------------------------------------------------+-------------------+----------------+---------------------+---------+----------------+---------------+---------+------------+-----------------------+------------+--------------------------------------+----------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+--------------------------------------+-----------+------+--------------------+------------------------+--------+----------+------------+----------------+ 1 row in set (0.00 sec) 2014-02-02 14:22:33,377 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (secstorage-1:ctx-61d52c4c) template 1 is already in store:16, type:Primary 2014-02-02 14:22:33,379 DEBUG [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-61d52c4c) Found template ro uting-1 in storage pool 16 with VMTemplateStoragePool id: 11868 2014-02-02 14:22:37,360 INFO [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-c605fca1) Unable to acqui re lock on VMTemplateStoragePool 11867 2014-02-02 14:22:49,487 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-b70366de) VmStatsCollector is running... 2014-02-02 14:24:09,967 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-f7f09a60) HostStatsCollector i s running... 2014-02-02 14:24:09,966 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-0ec2cf0e) StorageCollector is running... 2014-02-02 14:24:09,966 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-7:ctx-350f3bc0) Seq 2-760414210: Executing request 2014-02-02 14:23:58,075 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-333:ctx-b500c0a9) Seq 1-100250419 4: Executing request 2014-02-02 14:23:58,075 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-3763fe2a) Management se rver heartbeat takes too long to finish. profiler: Done. Duration: 75297ms, profilerHeartbeatUpdate: Done . Duration: 7251ms, profilerPeerScan: Started but not stopped 2014-02-02 14:24:34,629 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-61d52c4c) Cleaning up r esources for the vm VM[SecondaryStorageVm|s-2968-VM] in Starting state 2014-02-02 14:24:34,629 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-c605fca1) Cleaning up resources for the vm VM[ConsoleProxy|v-2-VM] in Starting state 2014-02-02 14:24:34,887 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-61d52c4c) Seq 1-1002523328: Sending { Cmd , MgmtId: 214076393326387, via: 1(xenserver-2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCo mmand":{"isProxy":false,"executeInSequence":false,"vmName":"s-2968-VM","wait":0}}] } 2014-02-02 14:24:34,887 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-61d52c4c) Seq 1-1002523328: Executing: { Cmd , MgmtId: 214076393326387, via: 1(xenserver-2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.Sto pCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-2968-VM","wait":0}}] } 2014-02-02 14:24:39,397 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c605fca1) Seq 2-760433604: Sending { Cmd , MgmtId: 214076393326387, via: 2(xenserver-1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopC ommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-2-VM","wait":0}}] } 2014-02-02 14:24:39,397 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c605fca1) Seq 2-760433604: Executing: { Cmd , MgmtId: 214076393326387, via: 2(xenserver-1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.St opCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-2-VM","wait":0}}] } 2014-02-02 14:24:39,414 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-b0e c7c97) Found 0 routers to update status. 2014-02-02 14:24:47,366 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-153:ctx-bd2d4c7f) Seq 1-100252332 8: Executing request 2014-02-02 14:24:47,366 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-275:ctx-d43e2414) Seq 2-760433604 : Executing request 2014-02-02 14:24:51,658 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-0ec2cf0e) Error trying to retr ieve storage stats java.lang.OutOfMemoryError: PermGen space 2014-02-02 14:24:59,201 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4ca44bf9) Unexpected exception when trying to execute queue item, java.lang.OutOfMemoryError: PermGen space 2014-02-02 14:24:59,215 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-7:ctx-350f3bc0) Seq 2-760414210: Response Received: 2014-02-02 14:24:59,215 DEBUG [c.c.a.t.Request] (DirectAgent-7:ctx-350f3bc0) Seq 2-760414210: Processing: { Ans: , MgmtId: 214076393326387, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } 2014-02-02 14:24:59,216 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-7:ctx-de693d5b) Seq 2-760414210: Executing request 2014-02-02 14:25:11,353 ERROR [c.c.s.StatsCollector] (StatsCollector-1:ctx-f7f09a60) Error trying to retrieve host stats java.lang.OutOfMemoryError: PermGen space 2014-02-02 14:25:39,109 DEBUG [o.a.c.n.c.m.ContrailManager] (DBSyncTimer:null) DB Sync task is running 2014-02-02 14:25:39,109 DEBUG [o.a.c.n.c.m.ServerDBSync] (DBSyncTimer:null) syncing cloudstack db with v -- This message was sent by Atlassian JIRA (v6.1.5#6160)