[
https://issues.apache.org/jira/browse/CLOUDSTACK-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13862083#comment-13862083
]
Sangeetha Hariharan commented on CLOUDSTACK-5446:
-------------------------------------------------
Is there a reason for leaving the primary store entries in the DB ? Should we
expect this entry to be cleared as well ?
> KVM-Secondary Store down-Even after secondary store is brought back up after
> being down for few hours,snapshot jobs do not get triggered with reason
> "there is other active snapshot tasks on the instance to which the volume is
> attached"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5446
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5446
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: Build from 4.3
> Reporter: Sangeetha Hariharan
> Assignee: Min Chen
> Fix For: 4.3.0
>
> Attachments: agentdown.rar, ssdown.rar
>
>
> KVM - Secondary Store down - Even after secondary store is brought back up
> after being down for few hours , snapshot jobs do not get triggered with
> reason "here is other active snapshot tasks on the instance to which the
> volume is attached, please try again later"
> Set up:
> Advanced Zone set up with 2 KVM (RHEL 6.3) hosts.
> Steps to reproduce the problem:
> 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we
> start with 10 Vms.
> 2. Start concurrent snapshots for ROOT volumes of all the Vms.
> 3. Shutdown the Secondary storage server when the snapshots are in the
> progress.
> 4. Bring the Secondary storage server up after ~ 12 hours.
> When the secondary storage was down:
> The snapshot jobs that were in progress timed out after 6 hours.
> Even after this , I do not see the hourly snapshots being scheduled due to
> the following reason:
>
> 2013-12-10 13:07:49,285 WARN [c.c.s.s.SnapshotSchedulerImpl]
> (SnapshotPollTask:ctx-cf0775f7) Scheduling snapshot failed due to
> com.cloud.utils.exception.CloudRuntimeException: There is other active
> snapshot tasks on the instance to which the volume is attached, please try
> again later
> Even after the secondary storage was brought up , there is no hourly
> snapshots being scheduled due to the same reason -
> "com.cloud.utils.exception.CloudRuntimeException: There is other active
> snapshot tasks on the instance to which the volume is attached, please try
> again later"
> mysql> select * FROM snapshots;
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | id | data_center_id | account_id | domain_id | volume_id | disk_offering_id
> | status | path | name |
> uuid | snapshot_type | type_description |
> size | created | removed | backup_snap_id | swift_id |
> sechost_id | prev_snap_id | hypervisor_type | version | s3_id |
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | 1 | 1 | 6 | 1 | 45 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131209234410 |
> ee2c46b7-7623-439a-9a30-63eec1d95c56 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 2 | 1 | 6 | 1 | 43 | 18
> | BackedUp | NULL | TestVM-1_ROOT-43_20131209234410 |
> 62c01389-49df-475b-b225-617d3903d25a | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 3 | 1 | 6 | 1 | 54 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131209234410 |
> 8c7621f0-5d94-4f22-b3f8-7954c810e1be | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 4 | 1 | 6 | 1 | 53 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131209234410 |
> 30c72de6-2a3e-4bf2-88da-b2a3b6ae9c66 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 5 | 1 | 6 | 1 | 52 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131209234411 |
> e8e9f99a-5a8b-4cfd-b743-327925955980 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 6 | 1 | 6 | 1 | 51 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131209234411 |
> 07e4edc8-b289-4f94-8696-ccf9db60e3ab | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 7 | 1 | 6 | 1 | 50 | 19
> | BackedUp | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131209234411 |
> b285aa1a-6650-4b90-9c2c-e8bf0c507bfb | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 8 | 1 | 6 | 1 | 49 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131209234412 |
> 31e37c8d-aab2-4ee6-9c43-c23501b08b02 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 9 | 1 | 6 | 1 | 48 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131209234412 |
> 60b6023e-138f-42c2-9867-f559e1d95352 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 10 | 1 | 6 | 1 | 47 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-2_ROOT-47_20131209234412 |
> fa37ebe5-f36c-4f19-b99f-249ab95c4e61 | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 11 | 1 | 6 | 1 | 46 | 18
> | BackedUp | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131209234412 |
> bd571765-b28e-433e-b480-ccdc3bc0450b | 3 | HOURLY |
> 21474836480 | 2013-12-09 23:44:12 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 12 | 1 | 6 | 1 | 45 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131210004410 |
> 4c6f50c2-661e-4bad-9f07-357b2ca4e7fc | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 13 | 1 | 6 | 1 | 43 | 18
> | CreatedOnPrimary | NULL | TestVM-1_ROOT-43_20131210004410 |
> bd3744ae-182f-4a0b-9685-5bef0c747365 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 14 | 1 | 6 | 1 | 54 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131210004410 |
> 14f53a2f-bb4b-40e5-aa90-e7b4be4a1076 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 15 | 1 | 6 | 1 | 53 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131210004410 |
> 88dd82f5-ade5-4dd5-a147-e0a35a15be85 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 16 | 1 | 6 | 1 | 52 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131210004410 |
> 1830af7d-97e5-488c-bc37-39c11961e509 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 17 | 1 | 6 | 1 | 51 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131210004410 |
> 3d452970-f042-4af6-a8cc-3003482da9a8 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 18 | 1 | 6 | 1 | 50 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131210004410 |
> 0cb1b176-bcfe-4bc2-9b5e-07ec8e9e97f5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:10 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 19 | 1 | 6 | 1 | 49 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131210004411 |
> 87337d8b-e698-47b1-961a-32479d44a6e5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 20 | 1 | 6 | 1 | 48 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131210004411 |
> c7b321bd-31b4-4c9c-be8f-c9231cb08530 | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 21 | 1 | 6 | 1 | 47 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-2_ROOT-47_20131210004411 |
> ea1a71b0-9b52-42e5-a252-7a3e8938bc0f | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 22 | 1 | 6 | 1 | 46 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131210004411 |
> f1fbc899-5f55-472c-b798-639350b2facd | 3 | HOURLY |
> 21474836480 | 2013-12-10 00:44:11 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 33 | 1 | 6 | 1 | 45 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-0_ROOT-45_20131210022749 |
> 396c80a8-299c-42b2-8c3d-32da5af9536c | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 34 | 1 | 6 | 1 | 43 | 18
> | CreatedOnPrimary | NULL | TestVM-1_ROOT-43_20131210022749 |
> 9444a355-08a6-4c7d-98f2-74729a0b28b5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 35 | 1 | 6 | 1 | 54 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-4_ROOT-54_20131210022749 |
> 09cbfc03-7f9f-470f-bc34-e2bbad72b9ba | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 36 | 1 | 6 | 1 | 53 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-3_ROOT-53_20131210022749 |
> 2e7259f0-82d2-4910-9047-617d54bda035 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 37 | 1 | 6 | 1 | 52 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-2_ROOT-52_20131210022749 |
> 420230ed-a037-4ed3-a019-031d24fd0bfb | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:49 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 38 | 1 | 6 | 1 | 51 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-1_ROOT-51_20131210022750 |
> 6bf87e88-3a14-467f-ac5b-ab620fd03206 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 39 | 1 | 6 | 1 | 50 | 19
> | CreatedOnPrimary | NULL | TestVM-tiny-host-1ps-0-0_ROOT-50_20131210022750 |
> 93021350-9df1-4234-9df2-869d9e9a912f | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 40 | 1 | 6 | 1 | 49 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-4_ROOT-49_20131210022750 |
> 4699ebdf-3093-4f2f-86b7-ae5c9195cc6a | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 41 | 1 | 6 | 1 | 48 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-3_ROOT-48_20131210022750 |
> 8d663979-31a2-48ad-884b-f1ef85a1f4d5 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:50 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 42 | 1 | 6 | 1 | 46 | 18
> | CreatedOnPrimary | NULL | TestVM-tiny-host-0ps-0-1_ROOT-46_20131210022751 |
> f80c2994-b612-4234-add4-27d323cf3dd3 | 3 | HOURLY |
> 21474836480 | 2013-12-10 02:27:51 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> +----+----------------+------------+-----------+-----------+------------------+------------------+------+-------------------------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> 32 rows in set (0.00 sec)
> mysql>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
