I upgraded my 4 host private dev cloud running on CentOS 6.4 x86_64 from
version 4.1.1 to 4.2 per upgrade instructions. I'm just using 4 HP Z 600
workstations (box 1 has two nics, the cloudstack-management server, the
mysql db, primary and secondary storage and a cloudstack-agent, and boxes 2
- 4 each just have primary storage and a cloudstack-agent). It's nothing
fancy, but it's been working perfectly now for months. All seemed to go
very smoothly except for the very last step:
{code}
nohup cloudstack-sysvmadm -d cs-east-dev1 -u root -p support -a > sysvm.log
2>&1 &
{code}
It could not restart the system vms for some reason. (I do not have the
original sysvm.log as I've now been playing with this failed upgrade for
two days). However, here is the sysvm.log from the very last attempt:
{code}
nohup: ignoring input
Stopping and starting 1 secondary storage vm(s)...
curl: (7) couldn't connect to host
ERROR: Failed to stop secondary storage vm with id 14
Done stopping and starting secondary storage vm(s)
Stopping and starting 0 console proxy vm(s)...
No running console proxy vms found
Stopping and starting 1 running routing vm(s)...
curl: (7) couldn't connect to host
2
Done restarting router(s).
{code}
As I mentioned above, I've been playing around with this for 2 days now and
actually got the 4.2 management server to finally start, but none of the
System VMs worked. I even restarted all of the CentOS hosts (which was a
huge hassle), but that didn't seem to help at all. I eventually found this
bug: https://issues.apache.org/jira/browse/CLOUDSTACK-4826 which seemed
similar to my issues.
I was also experiencing a strange issue where all 10 of my private
Management IP Addresses were used for some reason. Every time I restarted
the cloudstack-management service, 2 more IPs were taken up, but none ever
got released. Also since the System VMs would not start, my secondary
storage wouldn't start up either.
About an hour ago I gave up on 4.2 and I decided to roll back to 4.1.1 on
all 4 workstations. First I shutdown all VM instances, then stopped all
cloudstack-* services on all 4 workstations, and then ran a "yum downgrade
cloudstack-*" on all 4 workstations:
{code}
[root@cs-east-dev1 yum.repos.d]# yum downgrade cloudstack-*
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Downgrade Process
Loading mirror speeds from cached hostfile
* base: centos.mirror.nac.net
* extras: mirror.trouble-free.net
* rpmforge: mirror.us.leaseweb.net
* updates: mirror.cogentco.com
Resolving Dependencies
--> Running transaction check
---> Package cloudstack-agent.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-agent.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-awsapi.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-awsapi.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-cli.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-cli.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-common.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-common.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-management.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-management.x86_64 0:4.2.0-1.el6 will be erased
---> Package cloudstack-usage.x86_64 0:4.1.1-0.el6 will be a downgrade
---> Package cloudstack-usage.x86_64 0:4.2.0-1.el6 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================
Package Arch
Version Repository
Size
=============================================================================================================================================================
Downgrading:
cloudstack-agent x86_64
4.1.1-0.el6 cloudstack
37 M
cloudstack-awsapi x86_64
4.1.1-0.el6 cloudstack
56 M
cloudstack-cli x86_64
4.1.1-0.el6 cloudstack
32 k
cloudstack-common x86_64
4.1.1-0.el6 cloudstack
92 M
cloudstack-management x86_64
4.1.1-0.el6 cloudstack
55 M
cloudstack-usage x86_64
4.1.1-0.el6 cloudstack
37 M
Transaction Summary
=============================================================================================================================================================
Downgrade 6 Package(s)
Total download size: 278 M
Is this ok [y/N]: y
Downloading Packages:
(1/6): cloudstack-agent-4.1.1-0.el6.x86_64.rpm
| 37 MB
00:13
(2/6): cloudstack-awsapi-4.1.1-0.el6.x86_64.rpm
| 56 MB 01:01
(3/6): cloudstack-cli-4.1.1-0.el6.x86_64.rpm
| 32 kB
00:00
(4/6): cloudstack-common-4.1.1-0.el6.x86_64.rpm
| 92 MB 01:53
(5/6): cloudstack-management-4.1.1-0.el6.x86_64.rpm
| 55 MB 00:39
(6/6): cloudstack-usage-4.1.1-0.el6.x86_64.rpm
| 37 MB
00:12
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total
1.2 MB/s | 278 MB 04:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : cloudstack-common-4.1.1-0.el6.x86_64
1/12
Installing : cloudstack-management-4.1.1-0.el6.x86_64
2/12
warning: /etc/cloudstack/management/db.properties created as
/etc/cloudstack/management/db.properties.rpmnew
ln: creating symbolic link
`/usr/share/cloudstack-management/webapps7080/webapps': File exists
Please download vhd-util from
http://download.cloud.com.s3.amazonaws.com/tools/vhd-util and put it in
/usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/
Unable to determine ssl settings for server.xml, please run
cloudstack-setup-management manually
Unable to determine ssl settings for tomcat.conf, please run
cloudstack-setup-management manually
Installing : cloudstack-awsapi-4.1.1-0.el6.x86_64
3/12
Installing : cloudstack-agent-4.1.1-0.el6.x86_64
4/12
warning: /etc/cloudstack/agent/agent.properties created as
/etc/cloudstack/agent/agent.properties.rpmnew
Installing : cloudstack-cli-4.1.1-0.el6.x86_64
5/12
Installing : cloudstack-usage-4.1.1-0.el6.x86_64
6/12
Replacing db.properties with management server db.properties
Cleanup : cloudstack-awsapi-4.2.0-1.el6.x86_64
7/12
Cleanup : cloudstack-management-4.2.0-1.el6.x86_64
8/12
Stopping cloudstack-management:[ OK ]
Cleanup : cloudstack-agent-4.2.0-1.el6.x86_64
9/12
Stopping Cloud Agent:
Cleanup : cloudstack-common-4.2.0-1.el6.x86_64
10/12
Cleanup : cloudstack-cli-4.2.0-1.el6.x86_64
11/12
Cleanup : cloudstack-usage-4.2.0-1.el6.x86_64
12/12
Verifying : cloudstack-usage-4.1.1-0.el6.x86_64
1/12
Verifying : cloudstack-awsapi-4.1.1-0.el6.x86_64
2/12
Verifying : cloudstack-management-4.1.1-0.el6.x86_64
3/12
Verifying : cloudstack-cli-4.1.1-0.el6.x86_64
4/12
Verifying : cloudstack-common-4.1.1-0.el6.x86_64
5/12
Verifying : cloudstack-agent-4.1.1-0.el6.x86_64
6/12
Verifying : cloudstack-management-4.2.0-1.el6.x86_64
7/12
Verifying : cloudstack-cli-4.2.0-1.el6.x86_64
8/12
Verifying : cloudstack-common-4.2.0-1.el6.x86_64
9/12
Verifying : cloudstack-awsapi-4.2.0-1.el6.x86_64
10/12
Verifying : cloudstack-usage-4.2.0-1.el6.x86_64
11/12
Verifying : cloudstack-agent-4.2.0-1.el6.x86_64
12/12
Removed:
cloudstack-agent.x86_64 0:4.2.0-1.el6
cloudstack-awsapi.x86_64 0:4.2.0-1.el6
cloudstack-cli.x86_64 0:4.2.0-1.el6
cloudstack-common.x86_64 0:4.2.0-1.el6
cloudstack-management.x86_64 0:4.2.0-1.el6
cloudstack-usage.x86_64 0:4.2.0-1.el6
Installed:
cloudstack-agent.x86_64 0:4.1.1-0.el6
cloudstack-awsapi.x86_64 0:4.1.1-0.el6
cloudstack-cli.x86_64 0:4.1.1-0.el6
cloudstack-common.x86_64 0:4.1.1-0.el6
cloudstack-management.x86_64 0:4.1.1-0.el6
cloudstack-usage.x86_64 0:4.1.1-0.el6
Complete!
[root@cs-east-dev1 yum.repos.d]# service cloudstack-agent start
Starting Cloud Agent:
[root@cs-east-dev1 yum.repos.d]# service cloudstack-management start
Starting cloudstack-management: [ OK ]
[root@cs-east-dev1 yum.repos.d]
{code}
*NOTE*: On the other 3 hosts it only needed to downgrade cloudstack-agent
and cloudstack-common.
After the downgrade I received this silly error in the management-server.log
{code}
[root@cs-east-dev1 yum.repos.d]# tail -f
/var/log/cloudstack/management/management-server.log
2013-10-15 15:36:20,138 INFO [web.context.ContextLoader] (main:null) Root
WebApplicationContext: initialization started
2013-10-15 15:36:20,220 INFO [context.support.XmlWebApplicationContext]
(main:null) Refreshing Root WebApplicationContext: startup date [Tue Oct 15
15:36:20 EDT 2013]; root of context hierarchy
2013-10-15 15:36:20,276 INFO [factory.xml.XmlBeanDefinitionReader]
(main:null) Loading XML bean definitions from class path resource
[applicationContext.xml]
2013-10-15 15:36:20,713 INFO [factory.xml.XmlBeanDefinitionReader]
(main:null) Loading XML bean definitions from class path resource
[nonossComponentContext.xml]
2013-10-15 15:36:22,130 INFO
[factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:22,162 INFO [context.support.XmlWebApplicationContext]
(main:null) Bean 'transactionContextBuilder' of type [class
com.cloud.utils.db.TransactionContextBuilder] is not eligible for getting
processed by all BeanPostProcessors (for example: not eligible for
auto-proxying)
2013-10-15 15:36:22,167 INFO [context.support.XmlWebApplicationContext]
(main:null) Bean 'actionEventInterceptor' of type [class
com.cloud.event.ActionEventInterceptor] is not eligible for getting
processed by all BeanPostProcessors (for example: not eligible for
auto-proxying)
2013-10-15 15:36:22,212 INFO [factory.support.DefaultListableBeanFactory]
(main:null) Pre-instantiating singletons in
org.springframework.beans.factory.support.DefaultListableBeanFactory@757df1b6:
defining beans
[org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,componentContext,transactionContextBuilder,actionEventInterceptor,instantiatePostProcessor,onwireRegistry,messageSerializer,transportProvider,rpcProvider,eventBus,serviceOfferingDaoImpl,diskOfferingDaoImpl,dataCenterDaoImpl,hostPodDaoImpl,vlanDaoImpl,userDaoImpl,VMTemplateDaoImpl,hypervisorCapabilitiesDaoImpl,accountDaoImpl,accountDetailsDaoImpl,accountJoinDaoImpl,accountVlanMapDaoImpl,agentUpgradeDaoImpl,alertDaoImpl,asyncJobDaoImpl,asyncJobJoinDaoImpl,autoScalePolicyConditionMapDaoImpl,autoScalePolicyDaoImpl,autoScaleVmGroupDaoImpl,autoScaleVmGroupPolicyMapDaoImpl,autoScaleVmProfileDaoImpl,capacityDaoImpl,certificateDaoImpl,clusterDaoImpl,clusterDetailsDaoImpl,commandExecLogDaoImpl,conditionDaoImpl,consoleProxyDaoImpl,counterDaoImpl,dataCenterIpAddressDaoImpl,dataCenterJoinDaoImpl,dataCenterLinkLocalIpAddressDaoImpl,dataCenterVnetDaoImpl,dataStoreProviderDaoImpl,dcDetailsDaoImpl,engineDcDetailsDaoImpl,diskOfferingJoinDaoImpl,domainDaoImpl,domainRouterDaoImpl,domainRouterJoinDaoImpl,elasticLbVmMapDaoImpl,engineClusterDaoImpl,engineDataCenterDaoImpl,engineHostDaoImpl,engineHostPodDaoImpl,eventDaoImpl,eventJoinDaoImpl,externalFirewallDeviceDaoImpl,externalLoadBalancerDeviceDaoImpl,externalPublicIpStatisticsDaoImpl,firewallRulesCidrsDaoImpl,firewallRulesDaoImpl,guestOSCategoryDaoImpl,guestOSDaoImpl,highAvailabilityDaoImpl,hostDaoImpl,hostDetailsDaoImpl,engineHostDetailsDaoImpl,hostJoinDaoImpl,engineHostTagsDaoImpl,hostTagsDaoImpl,hostTransferMapDaoImpl,iPAddressDaoImpl,identityDaoImpl,imageDaoStoreDaoImpl,imageDataDaoImpl,imageDataStoreProviderDaoImpl,inlineLoadBalancerNicMapDaoImpl,instanceGroupDaoImpl,instanceGroupJoinDaoImpl,instanceGroupVMMapDaoImpl,itWorkDaoImpl,keystoreDaoImpl,lBStickinessPolicyDaoImpl,launchPermissionDaoImpl,loadBalancerDaoImpl,loadBalancerVMMapDaoImpl,managementServerHostDaoImpl,managementServerHostPeerDaoImpl,networkAccountDaoImpl,networkDaoImpl,networkDomainDaoImpl,networkExternalFirewallDaoImpl,networkExternalLoadBalancerDaoImpl,networkOfferingDaoImpl,networkOfferingServiceMapDaoImpl,networkOpDaoImpl,networkRuleConfigDaoImpl,networkServiceMapDaoImpl,nicDaoImpl,objectInDataStoreDaoImpl,ovsTunnelInterfaceDaoImpl,ovsTunnelNetworkDaoImpl,physicalNetworkDaoImpl,physicalNetworkIsolationMethodDaoImpl,physicalNetworkServiceProviderDaoImpl,physicalNetworkTagDaoImpl,physicalNetworkTrafficTypeDaoImpl,podVlanDaoImpl,podVlanMapDaoImpl,portForwardingRulesDaoImpl,primaryDataStoreDaoImpl,primaryDataStoreDetailsDaoImpl,privateIpDaoImpl,projectAccountDaoImpl,projectAccountJoinDaoImpl,projectDaoImpl,projectInvitationDaoImpl,projectInvitationJoinDaoImpl,projectJoinDaoImpl,regionDaoImpl,remoteAccessVpnDaoImpl,resourceCountDaoImpl,resourceLimitDaoImpl,resourceTagJoinDaoImpl,resourceTagsDaoImpl,routerNetworkDaoImpl,s3DaoImpl,sSHKeyPairDaoImpl,secondaryStorageVmDaoImpl,securityGroupDaoImpl,securityGroupJoinDaoImpl,securityGroupRuleDaoImpl,securityGroupRulesDaoImpl,securityGroupVMMapDaoImpl,securityGroupWorkDaoImpl,serviceOfferingJoinDaoImpl,site2SiteCustomerGatewayDaoImpl,site2SiteVpnConnectionDaoImpl,site2SiteVpnGatewayDaoImpl,snapshotDao2Impl,snapshotDaoImpl,snapshotPolicyDaoImpl,snapshotScheduleDaoImpl,stackMaidDaoImpl,staticRouteDaoImpl,storageNetworkIpAddressDaoImpl,storageNetworkIpRangeDaoImpl,storagePoolDaoImpl,storagePoolDetailsDaoImpl,storagePoolHostDaoImpl,storagePoolJoinDaoImpl,storagePoolWorkDaoImpl,swiftDaoImpl,syncQueueDaoImpl,syncQueueItemDaoImpl,templatePrimaryDataStoreDaoImpl,uploadDaoImpl,usageDaoImpl,usageEventDaoImpl,usageIPAddressDaoImpl,usageJobDaoImpl,usageLoadBalancerPolicyDaoImpl,usageNetworkDaoImpl,usageNetworkOfferingDaoImpl,usagePortForwardingRuleDaoImpl,usageSecurityGroupDaoImpl,usageStorageDaoImpl,usageVMInstanceDaoImpl,usageVPNUserDaoImpl,usageVolumeDaoImpl,userAccountDaoImpl,userAccountJoinDaoImpl,userIpv6AddressDaoImpl,userStatisticsDaoImpl,userStatsLogDaoImpl,userVmDetailsDaoImpl,userVmJoinDaoImpl,vMComputeTagDaoImpl,vMEntityDaoImpl,vMInstanceDaoImpl,vMNetworkMapDaoImpl,vMReservationDaoImpl,vMRootDiskTagDaoImpl,vMTemplateDetailsDaoImpl,vMTemplateHostDaoImpl,vMTemplatePoolDaoImpl,vMTemplateS3DaoImpl,vMTemplateSwiftDaoImpl,vMTemplateZoneDaoImpl,versionDaoImpl,virtualRouterProviderDaoImpl,vmRulesetLogDaoImpl,volumeDao2Impl,volumeDaoImpl,volumeHostDaoImpl,volumeJoinDaoImpl,volumeReservationDaoImpl,vpcDaoImpl,vpcGatewayDaoImpl,vpcOfferingDaoImpl,vpcOfferingServiceMapDaoImpl,vpnUserDaoImpl,userVmDaoImpl,portProfileDaoImpl,clusterVSMMapDaoImpl,encryptionSecretKeyChecker,StaticRoleBasedAPIAccessChecker,databaseIntegrityChecker,domainChecker,basicAgentAuthManager,MD5UserAuthenticator,LDAPUserAuthenticator,PlainTextUserAuthenticator,Ovs,SecurityGroupProvider,VirtualRouter,VpcVirtualRouter,elasticLoadBalancerElement,firstFitAllocator,randomAllocator,FirstFitRouting,LocalStoragePoolAllocator,FirstFitStoragePoolAllocator,UserConcentratedAllocator,useLocalForRootAllocator,garbageCollectingStoragePoolAllocator,randomStoragePoolAllocator,recreatableFencer,recreateHostAllocator,secondaryStorageVmDefaultAllocator,ConsoleProxyAllocator,ExteralIpAddressAllocator,hyervisorTemplateAdapter,clusterAlertAdapter,consoleProxyAlertAdapter,secondaryStorageVmAlertAdapter,clusterServiceServletAdapter,CheckOnAgentInvestigator,XenServerInvestigator,UserVmDomRInvestigator,ManagementIPSystemVMInvestigator,XenServerFencer,KVMFencer,OvmFencer,XcpServerDiscoverer,SecondaryStorageDiscoverer,KvmServerDiscoverer,HypervServerDiscoverer,OvmDiscoverer,dummyHostDiscoverer,UserDispersingPlanner,UserConcentratedPodPlanner,clusterBasedAgentLoadBalancerPlanner,StorageNetworkGuru,ExternalGuestNetworkGuru,PublicNetworkGuru,PodBasedNetworkGuru,ControlNetworkGuru,DirectNetworkGuru,DirectPodBasedNetworkGuru,OvsGuestNetworkGuru,PrivateNetworkGuru,XenServerGuru,KVMGuru,OvmGuru,HypervisorPlannerSelector,accountManagerImpl,managementServerImpl,storageManagerImpl,FirstFitPlanner,resourceManagerImpl,highAvailabilityManagerExtImpl,clusteredVirtualMachineManagerImpl,networkManagerImpl,networkModelImpl,configurationServerImpl,clusterManagerImpl,clusteredAgentManagerImpl,userVmManagerImpl,consoleProxyManagerImpl,securityGroupManagerImpl2,premiumSecondaryStorageManagerImpl,ipv6AddressManagerImpl,apiRateLimitServiceImpl,alertManagerImpl,asyncJobExecutorContextImpl,asyncJobManagerImpl,autoScaleManagerImpl,capacityManagerImpl,clusterFenceManagerImpl,configurationManagerImpl,dataStoreProviderManagerImpl,domainManagerImpl,downloadMonitorImpl,elasticLoadBalancerManagerImpl,entityManagerImpl,externalLoadBalancerUsageManagerImpl,externalNetworkDeviceManagerImpl,firewallManagerImpl,hypervisorGuruManagerImpl,identityServiceImpl,keystoreManagerImpl,loadBalancingRulesManagerImpl,networkACLManagerImpl,networkServiceImpl,networkUsageManagerImpl,oCFS2ManagerImpl,ovsTunnelManagerImpl,projectManagerImpl,queryManagerImpl,regionManagerImpl,regionServiceImpl,remoteAccessVpnManagerImpl,resourceLimitManagerImpl,rulesManagerImpl,s3ManagerImpl,site2SiteVpnManagerImpl,snapshotManagerImpl,snapshotSchedulerImpl,storageNetworkManagerImpl,swiftManagerImpl,syncQueueManagerImpl,taggedResourceManagerImpl,templateManagerImpl,upgradeManagerImpl,uploadMonitorImpl,usageServiceImpl,virtualNetworkApplianceManagerImpl,vpcManagerImpl,vpcVirtualNetworkApplianceManagerImpl,cloudZonesStartupProcessor,managementServerNode,actionEventUtils,agentMonitor,alertGenerator,apiDBUtils,apiDiscoveryServiceImpl,apiDispatcher,apiResponseHelper,apiServer,apiServlet,cloudOrchestrator,clusterRestService,consoleProxyServlet,dataCenterResourceManagerImpl,dataDisk,dataMotionServiceImpl,dataObjectManagerImpl,dataStoreManagerImpl,defaultEndPointSelector,defaultImageDataStoreProvider,defaultImageMotionStrategy,defaultPrimaryDataStoreProviderManagerImpl,defaultPrimaryDatastoreProviderImpl,hypervisorBasedSnapshot,hypervsiorHostEndPointRpcServer,iSCSI,ISO,imageDataFactoryImpl,imageDataManagerImpl,imageDataStoreHelper,imageDataStoreManagerImpl,imageFormatHelper,imageMotionServiceImpl,imageServiceImpl,iso,networkFileSystem,networkRestService,OVA,objectInDataStoreManagerImpl,podRestService,primaryDataStoreHelper,provisioningServiceImpl,QCOW2,registerCompleteServlet,rootDisk,snapshotDataFactoryImpl,snapshotServiceImpl,statsCollector,templateInstallStrategyImpl,unknown,usageEventUtils,userContextInitializer,VHD,vMEntityManagerImpl,virtualMachineEntityImpl,virtualMachineRestService,volumeDataFactoryImpl,volumeManagerImpl,volumeRestService,volumeServiceImpl,volumeTypeHelper,zoneRestService,premiumDatabaseUpgradeChecker,configurationDaoImpl,VmwareFencer,vmwareServerDiscoverer,VMwareGuru,VmwareInvestigator,VmwareManager,vmwareContextFactory,niciraNvpDaoImpl,niciraNvpNicMappingDaoImpl,niciraNvpRouterMappingDaoImpl,NiciraNvpGuestNetworkGuru,NiciraNvp,lunDaoImpl,poolDaoImpl,netappVolumeDaoImpl,NetappManager,JuniperSRX,netScalerPodDaoImpl,Netscaler,F5BigIP,ciscoNexusVSMDeviceDaoImpl,CiscoNexus1000vVSM,bigSwitchVnsDaoImpl,bigSwitchVnsElement,userAuthenticators,securityCheckers,resourceDiscoverers,haInvestigators,haFenceBuilders,deploymentPlanners,podAllocators,hostAllocators,storagePoolAllocators,ipDeployers,dhcpProviders,networkGurus,networkElements,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0];
root of factory hierarchy
2013-10-15 15:36:22,219 INFO [utils.component.ComponentContext]
(main:null) Setup Spring Application context
2013-10-15 15:36:22,515 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
(main:null) Encryption Type: file
2013-10-15 15:36:27,126 INFO [cloud.serializer.GsonHelper] (main:null)
Default Builder inited.
2013-10-15 15:36:27,901 INFO [hypervisor.vmware.VmwareServerDiscoverer]
(main:null) VmwareServerDiscoverer is constructed
2013-10-15 15:36:28,117 INFO [web.context.ContextLoader] (main:null) Root
WebApplicationContext: initialization completed in 7978 ms
2013-10-15 15:36:28,134 INFO [cloud.utils.LogUtils] (main:null) log4j
configuration found at /etc/cloudstack/management/log4j-cloud.xml
2013-10-15 15:36:28,151 INFO
[factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:28,152 INFO
[factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:28,153 INFO
[factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null)
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2013-10-15 15:36:33,514 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackAccountDaoImpl_EnhancerByCloudStack_3de75b6e
2013-10-15 15:36:33,516 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.OfferingDaoImpl_EnhancerByCloudStack_d2f1cac8
2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SMetaDaoImpl_EnhancerByCloudStack_ef98bac0
2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.UserCredentialsDaoImpl_EnhancerByCloudStack_909a85df
2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MultipartMetaDaoImpl_EnhancerByCloudStack_d7013d7f
2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackConfigurationDaoImpl_EnhancerByCloudStack_fecccd9d
2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.BucketPolicyDaoImpl_EnhancerByCloudStack_56673762
2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SObjectItemDaoImpl_EnhancerByCloudStack_3a79ef2b
2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MHostMountDaoImpl_EnhancerByCloudStack_8f7d233a
2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MHostDaoImpl_EnhancerByCloudStack_165457b
2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MultiPartUploadsDaoImpl_EnhancerByCloudStack_bcd9eaa8
2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SBucketDaoImpl_EnhancerByCloudStack_8e162773
2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SAclDaoImpl_EnhancerByCloudStack_104ffef
2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.MultiPartPartsDaoImpl_EnhancerByCloudStack_88d4152a
2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackSvcOfferingDaoImpl_EnhancerByCloudStack_882a018d
2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_29f7bbc5
2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStack_c20b110
2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_10d22bb6
2013-10-15 15:36:33,520 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_635f1066
2013-10-15 15:36:33,561 INFO [utils.component.ComponentContext]
(Timer-1:null) Running SystemIntegrityChecker encryptionSecretKeyChecker
2013-10-15 15:36:33,563 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
(Timer-1:null) Encryption Type: file
2013-10-15 15:36:33,563 WARN [utils.crypt.EncryptionSecretKeyChecker]
(Timer-1:null) Encryption already enabled, is check() called twice?
2013-10-15 15:36:33,564 INFO [utils.component.ComponentContext]
(Timer-1:null) Running SystemIntegrityChecker databaseIntegrityChecker
2013-10-15 15:36:33,564 INFO [cloud.upgrade.DatabaseIntegrityChecker]
(Timer-1:null) Grabbing lock to check for database integrity.
2013-10-15 15:36:33,750 INFO [cloud.upgrade.DatabaseIntegrityChecker]
(Timer-1:null) Performing database integrity check
2013-10-15 15:36:33,754 INFO [utils.component.ComponentContext]
(main:null) Configuring
com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloudStack_8e8faadb
2013-10-15 15:36:33,763 DEBUG [cloud.upgrade.DatabaseIntegrityChecker]
(Timer-1:null) No duplicate hosts with the same local storage found in
database
2013-10-15 15:36:33,767 DEBUG [upgrade.dao.VersionDaoImpl] (Timer-1:null)
Checking to see if the database is at a version before it was the version
table is created
2013-10-15 15:36:33,787 INFO [utils.component.ComponentContext]
(Timer-1:null) Running SystemIntegrityChecker managementServerNode
2013-10-15 15:36:33,788 ERROR [utils.component.ComponentContext]
(Timer-1:null) *System integrity check failed. Refuse to startup*
[root@cs-east-dev1 yum.repos.d]#
{code}
I googled this issue and found the following discussion:
http://mail-archives.apache.org/mod_mbox/cloudstack-issues/201309.mbox/%3CJIRA.12671428.1380572215050.5987.1380572367958@arcas%3E
and this discussion points to the following defects:
https://issues.apache.org/jira/browse/CLOUDSTACK-4770 &&
https://issues.apache.org/jira/browse/CLOUDSTACK-4170
But defect CLOUDSTACK-4770 is set to *minor *and there doesn't seem to be
any activity on it.
In the CLOUDSTACK-4770 defect report Richard Chatterton mentions having a
MAC address of 00:00:00:00:00:00, so I performed the same steps as Richard
(except for version 4.1.1) and got the following output:
{code}
[root@cs-east-dev1 ~]# java -classpath
/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/cloud-utils-4.1.1.jar
com.cloud.utils.net.MacAddress
*addr in integer is 0*
*addr in bytes is 0 0 0 0 0 0*
*addr in char is 00:00:00:00:00:00*
[root@cs-east-dev1 ~]# ifconfig -a
*cloud0 Link encap:Ethernet HWaddr 00:00:00:00:00:00*
inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::9c89:feff:feca:45e5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:1752 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:73800 (72.0 KiB)
cloudbr0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A
inet addr:10.97.36.115 Bcast:10.97.36.255 Mask:255.255.255.0
inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:36186 errors:0 dropped:0 overruns:0 frame:0
TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3830135 (3.6 MiB) TX bytes:746 (746.0 b)
eth0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A
inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:44652 errors:0 dropped:0 overruns:0 frame:0
TX packets:251 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5815431 (5.5 MiB) TX bytes:19286 (18.8 KiB)
Interrupt:17
eth1 Link encap:Ethernet HWaddr 00:18:F8:0B:53:2B
inet addr:10.97.36.105 Bcast:10.97.36.255 Mask:255.255.255.0
inet6 addr: fe80::218:f8ff:fe0b:532b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:542135 errors:0 dropped:27 overruns:0 frame:0
TX packets:422950 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:506474573 (483.0 MiB) TX bytes:157977312 (150.6 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:547879 errors:0 dropped:0 overruns:0 frame:0
TX packets:547879 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:122259530 (116.5 MiB) TX bytes:122259530 (116.5 MiB)
virbr0 Link encap:Ethernet HWaddr 52:54:00:40:5C:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
virbr0-nic Link encap:Ethernet HWaddr 52:54:00:40:5C:17
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
[root@cs-east-dev1 ~]#
{code}
So... then I was curious what would happen if I simply restarted the
network service:
{code}
[root@cs-east-dev1 ~]# service network status
Configured devices:
lo cloudbr0 eth0 eth1
Currently active devices:
lo eth0 eth1 cloudbr0 virbr0 cloud0
[root@cs-east-dev1 ~]# service network restart
Shutting down interface cloudbr0: [ OK ]
Shutting down interface eth0: [ OK ]
Shutting down interface eth1: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]
Bringing up interface eth1:
Determining IP information for eth1... done.
[ OK ]
Bringing up interface cloudbr0: [ OK ]
[root@cs-east-dev1 ~]# ifconfig -a
cloudbr0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A
inet addr:10.97.36.115 Bcast:10.97.36.255 Mask:255.255.255.0
inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:30 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2639 (2.5 KiB) TX bytes:566 (566.0 b)
cloud0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::9c89:feff:feca:45e5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:1752 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:73800 (72.0 KiB)
eth0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A
inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:65 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7266 (7.0 KiB) TX bytes:1132 (1.1 KiB)
Interrupt:17
eth1 Link encap:Ethernet HWaddr 00:18:F8:0B:53:2B
inet addr:10.97.36.105 Bcast:10.97.36.255 Mask:255.255.255.0
inet6 addr: fe80::218:f8ff:fe0b:532b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:546735 errors:0 dropped:27 overruns:0 frame:0
TX packets:428375 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:506928715 (483.4 MiB) TX bytes:160011479 (152.5 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:548153 errors:0 dropped:0 overruns:0 frame:0
TX packets:548153 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:122274810 (116.6 MiB) TX bytes:122274810 (116.6 MiB)
virbr0 Link encap:Ethernet HWaddr 52:54:00:40:5C:17
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
virbr0-nic Link encap:Ethernet HWaddr 52:54:00:40:5C:17
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
[root@cs-east-dev1 ~]#
{code}
Notice how now cloudbr0 is listed first, and that has a valid MAC address.
Believe it or not, now the management server starts up fine, however... the
System VMs still wouldn't start. They were stuck in the "Starting" state.
So... I downloaded the System ISO seed again
{code}
[root@cs-east-dev1 ~]#
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
-m /export/secondary -u
http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2-h
kvm -F
--2013-10-15 16:31:45--
http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2
Resolving download.cloud.com... 207.171.187.118
Connecting to download.cloud.com|207.171.187.118|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 286036668 (273M) [binary/octet-stream]
Saving to:
â/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2â
100%[===================================================================================================================>]
286,036,668 4.15M/s in 73s
2013-10-15 16:32:59 (3.73 MB/s) -
â/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2â
saved [286036668/286036668]
Uncompressing to
/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2.tmp
(type bz2)...could take a long time
Moving to
/export/secondary/template/tmpl/1/3///1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2...could
take a while
Successfully installed system VM template to
/export/secondary/template/tmpl/1/3/
[root@cs-east-dev1 ~]#
{code}
Once that was complete I destroyed the system VMs and restarted
cloudstack-management. Now everything seems back to normal.
Bottom line, I don't feel that 4.2 is production ready yet for KVM on
CentOS 6.4. I hope this helps someone else...
--
Best Regards,
Adam Scarcella
Sr. Support Engineer | Spotfire, TIBCO Software Inc.
Web: http://spotfire.tibco.com/support
