Chandan Purushothama created CLOUDSTACK-4305: ------------------------------------------------
Summary: ASF 4.1 to 4.2 Upgrade: Unable to upgrade the database - Failed to update volume format to RAW for volumes on RBD pools Key: CLOUDSTACK-4305 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4305 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Upgrade Affects Versions: 4.2.0 Reporter: Chandan Purushothama Priority: Blocker Fix For: 4.2.0 Steps to Reproduce: 1. Upgrade 4.1 to 4.2 Setup. Observations: 2013-08-13 14:28:54,529 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating System Vm template IDs 2013-08-13 14:28:54,546 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating XenServer System Vms 2013-08-13 14:28:54,589 WARN [upgrade.dao.Upgrade412to420] (Timer-2:null) 4.2.0 XenServer SystemVm template not found. XenServer hypervisor is not used, so not failing upgrade 2013-08-13 14:28:54,602 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating VMware System Vms 2013-08-13 14:28:54,605 WARN [upgrade.dao.Upgrade412to420] (Timer-2:null) 4.2.0 VMware SystemVm template not found. VMware hypervisor is not used, so not failing upgrade 2013-08-13 14:28:54,607 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating Hyperv System Vms 2013-08-13 14:28:54,607 WARN [upgrade.dao.Upgrade412to420] (Timer-2:null) 4.2.0 Hyperv SystemVm template not found. Hyperv hypervisor is not used, so not failing upgrade 2013-08-13 14:28:54,610 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating LXC System Vms 2013-08-13 14:28:54,610 WARN [upgrade.dao.Upgrade412to420] (Timer-2:null) 4.2.0 LXC SystemVm template not found. LXC hypervisor is not used, so not failing upgrade 2013-08-13 14:28:54,611 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating KVM System Vms 2013-08-13 14:28:54,612 WARN [upgrade.dao.Upgrade412to420] (Timer-2:null) 4.2.0 KVM SystemVm template not found. KVM hypervisor is not used, so not failing upgrade 2013-08-13 14:28:54,616 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating System Vm Template IDs Complete 2013-08-13 14:28:54,700 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating network ACLs 2013-08-13 14:28:54,701 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Done updating network ACLs 2013-08-13 14:28:54,701 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Checking if host_details index exists, if not we will add it 2013-08-13 14:28:54,792 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Index did not exist on host_details - added new one 2013-08-13 14:28:54,795 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating KVM snapshots 2013-08-13 14:28:54,796 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Done updating KVM snapshots 2013-08-13 14:28:54,796 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Verify and set the KVM snapshot flag if snapshot was used. 2013-08-13 14:28:54,798 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Done set KVM snapshot flag. 2013-08-13 14:28:54,798 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Dropping index i_alert__last_sent if it exists 2013-08-13 14:28:54,886 DEBUG [upgrade.dao.DbUpgradeUtils] (Timer-2:null) Key last_sent is dropped successfully from the table alert 2013-08-13 14:28:54,926 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Added index i_alert__last_sent for table alert 2013-08-13 14:28:55,434 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Added foreign keys for table baremetal_dhcp_devices 2013-08-13 14:28:55,552 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Added foreign keys for table baremetal_pxe_devices 2013-08-13 14:28:55,552 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating secondary storage to image store 2013-08-13 14:28:55,552 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Checking if we need to migrate NFS secondary storage to image store or staging store 2013-08-13 14:28:55,559 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating NFS secondary storage to Image store 2013-08-13 14:28:55,570 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed migrating secondary storage to image store 2013-08-13 14:28:55,571 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating volume_store_ref table from volume_host_ref table 2013-08-13 14:28:55,573 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Insert modified 0 rows 2013-08-13 14:28:55,573 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Update modified 0 rows 2013-08-13 14:28:55,573 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed updating volume_store_ref table from volume_host_ref table 2013-08-13 14:28:55,574 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating template_store_ref table from template_host_ref table 2013-08-13 14:28:55,580 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Insert modified 0 rows 2013-08-13 14:28:55,580 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Update modified 0 rows 2013-08-13 14:28:55,581 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed updating template_store_ref table from template_host_ref table 2013-08-13 14:28:55,581 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating snapshot_store_ref table from snapshots table 2013-08-13 14:28:55,582 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Inserted 0 snapshots into snapshot_store_ref 2013-08-13 14:28:55,588 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Inserted 0 KVM snapshots into snapshot_store_ref 2013-08-13 14:28:55,588 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed updating snapshot_store_ref table from snapshots table 2013-08-13 14:28:55,589 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating S3 to image store 2013-08-13 14:28:55,593 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating template_s3_ref to template_store_ref 2013-08-13 14:28:55,594 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating template_store_ref table from template_s3_ref table 2013-08-13 14:28:55,615 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed migrating template_s3_ref table. 2013-08-13 14:28:55,615 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating s3 backedup snapshots to snapshot_store_ref 2013-08-13 14:28:55,615 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating snapshot_store_ref table from snapshots table for s3 2013-08-13 14:28:55,618 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed updating snapshot_store_ref table from s3 snapshots entries 2013-08-13 14:28:55,619 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed migrating S3 secondary storage to image store 2013-08-13 14:28:55,619 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating Swift to image store 2013-08-13 14:28:55,620 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating template_swift_ref to template_store_ref 2013-08-13 14:28:55,625 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating template_store_ref table from template_swift_ref table 2013-08-13 14:28:55,626 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed migrating template_swift_ref table. 2013-08-13 14:28:55,626 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Migrating swift backedup snapshots to snapshot_store_ref 2013-08-13 14:28:55,627 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Updating snapshot_store_ref table from snapshots table for swift 2013-08-13 14:28:55,634 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed updating snapshot_store_ref table from swift snapshots entries 2013-08-13 14:28:55,634 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Completed migrating Swift secondary storage to image store 2013-08-13 14:28:55,634 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Dropping foreign key fk_nicira_nvp_nic_map__nic from the table nicira_nvp_nic_map if it exists 2013-08-13 14:28:55,709 DEBUG [upgrade.dao.DbUpgradeUtils] (Timer-2:null) Key fk_nicira_nvp_nic_map__nic is dropped successfully from the table nicira_nvp_nic_map 2013-08-13 14:28:55,744 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Added foreign key fk_nicira_nvp_nic_map__nic to the table nicira_nvp_nic_map 2013-08-13 14:28:55,744 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Dropping foreign key fk_router_network_ref__router_id from the table router_network_ref if it exists 2013-08-13 14:28:55,785 DEBUG [upgrade.dao.DbUpgradeUtils] (Timer-2:null) Key fk_router_network_ref__router_id is dropped successfully from the table router_network_ref 2013-08-13 14:28:55,812 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Added foreign key fk_router_network_ref__router_id to the table router_network_ref 2013-08-13 14:28:55,888 DEBUG [upgrade.dao.Upgrade412to420] (Timer-2:null) Setting format to RAW for all volumes on RBD primary storage pools 2013-08-13 14:28:55,894 ERROR [cloud.upgrade.DatabaseUpgradeChecker] (Timer-2:null) Unable to upgrade the database com.cloud.utils.exception.CloudRuntimeException: Failed to update volume format to RAW for volumes on RBD pools at com.cloud.upgrade.dao.Upgrade412to420.setRAWformatForRBDVolumes(Upgrade412to420.java:2624) at com.cloud.upgrade.dao.Upgrade412to420.performDataMigration(Upgrade412to420.java:111) at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:302) at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:427) at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:90) at com.cloud.servlet.CloudStartupServlet$1.run(CloudStartupServlet.java:54) at java.util.TimerThread.mainLoop(Timer.java:534) at java.util.TimerThread.run(Timer.java:484) 2013-08-13 14:28:55,896 DEBUG [db.Transaction.Transaction] (Timer-2:null) Rolling back the transaction: Time = 11246 Name = -CloudStartupServlet$1.run:52-TimerThread.mainLoop:534-TimerThread.run:484; called by -Transaction.rollback:898-Transaction.removeUpTo:841-Transaction.close:665-DatabaseUpgradeChecker.upgrade:331-DatabaseUpgradeChecker.check:427-ComponentContext.initComponentsLifeCycle:90-CloudStartupServlet$1.run:54-TimerThread.mainLoop:534-TimerThread.run:484 2013-08-13 14:28:55,901 ERROR [utils.component.ComponentContext] (Timer-2:null) System integrity check failed. Refuse to startup com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the database at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:329) at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:427) at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:90) at com.cloud.servlet.CloudStartupServlet$1.run(CloudStartupServlet.java:54) at java.util.TimerThread.mainLoop(Timer.java:534) at java.util.TimerThread.run(Timer.java:484) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to update volume format to RAW for volumes on RBD pools at com.cloud.upgrade.dao.Upgrade412to420.setRAWformatForRBDVolumes(Upgrade412to420.java:2624) at com.cloud.upgrade.dao.Upgrade412to420.performDataMigration(Upgrade412to420.java:111) at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:302) ... 5 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira