Hi Adam, I also run into the same issues with upgrade from 4.0.0 to 4.2, on test CS installation, and was really pissed off, because of documentation is kind of incomplete on the upgrade process, or at least really buggy.. but I managed to resolve it for now !
1. So if you want to try upgrade again (system-vm not starting issue), do it as per docs, and then folow this guy's post, I did, and it worked very well... http://cloud.kelceydamage.com/cloudfire/blog/2013/10/08/conquering-the-cloudstack-4-2-dragon-kvm/(I used method 2) 2. BTW, than problem of management IP being used (assigned to system VM that were not working/starting, and after you destroy the system VM, it's IP addresses still show in database) - the solution for this is the following: Inside "cloud" database, table "op_dc_ip_address_alloc", delete values (reset to NULL) of the folowing fields (for all rows/IP addresses that you know are NOT assigned to any VM): "nic_id", "reservation_id" "taken". After that, restart management server, and it will be all fine. I do agree that CS 4.2 on CentOS is not production ready... On 16 October 2013 07:53, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > H Adam, > > forgive me if I missed some important clue in your report. It seems to > me that it is the upgrade procedure that is not production ready, not > version 4.2. Am I right? > > regards, > Daan > > On Tue, Oct 15, 2013 at 11:10 PM, Adam <adam.scarce...@gmail.com> wrote: > > I upgraded my 4 host private dev cloud running on CentOS 6.4 x86_64 from > > version 4.1.1 to 4.2 per upgrade instructions. I'm just using 4 HP Z 600 > > workstations (box 1 has two nics, the cloudstack-management server, the > > mysql db, primary and secondary storage and a cloudstack-agent, and > boxes 2 > > - 4 each just have primary storage and a cloudstack-agent). It's nothing > > fancy, but it's been working perfectly now for months. All seemed to go > > very smoothly except for the very last step: > > > > {code} > > nohup cloudstack-sysvmadm -d cs-east-dev1 -u root -p support -a > > sysvm.log > > 2>&1 & > > {code} > > > > > > It could not restart the system vms for some reason. (I do not have the > > original sysvm.log as I've now been playing with this failed upgrade for > > two days). However, here is the sysvm.log from the very last attempt: > > > > {code} > > nohup: ignoring input > > > > Stopping and starting 1 secondary storage vm(s)... > > curl: (7) couldn't connect to host > > ERROR: Failed to stop secondary storage vm with id 14 > > > > Done stopping and starting secondary storage vm(s) > > > > Stopping and starting 0 console proxy vm(s)... > > No running console proxy vms found > > > > > > Stopping and starting 1 running routing vm(s)... > > curl: (7) couldn't connect to host > > 2 > > Done restarting router(s). > > {code} > > > > > > As I mentioned above, I've been playing around with this for 2 days now > and > > actually got the 4.2 management server to finally start, but none of the > > System VMs worked. I even restarted all of the CentOS hosts (which was a > > huge hassle), but that didn't seem to help at all. I eventually found > this > > bug: https://issues.apache.org/jira/browse/CLOUDSTACK-4826 which seemed > > similar to my issues. > > > > I was also experiencing a strange issue where all 10 of my private > > Management IP Addresses were used for some reason. Every time I restarted > > the cloudstack-management service, 2 more IPs were taken up, but none > ever > > got released. Also since the System VMs would not start, my secondary > > storage wouldn't start up either. > > > > About an hour ago I gave up on 4.2 and I decided to roll back to 4.1.1 on > > all 4 workstations. First I shutdown all VM instances, then stopped all > > cloudstack-* services on all 4 workstations, and then ran a "yum > downgrade > > cloudstack-*" on all 4 workstations: > > > > {code} > > [root@cs-east-dev1 yum.repos.d]# yum downgrade cloudstack-* > > Loaded plugins: fastestmirror, refresh-packagekit, security > > Setting up Downgrade Process > > Loading mirror speeds from cached hostfile > > * base: centos.mirror.nac.net > > * extras: mirror.trouble-free.net > > * rpmforge: mirror.us.leaseweb.net > > * updates: mirror.cogentco.com > > Resolving Dependencies > > --> Running transaction check > > ---> Package cloudstack-agent.x86_64 0:4.1.1-0.el6 will be a downgrade > > ---> Package cloudstack-agent.x86_64 0:4.2.0-1.el6 will be erased > > ---> Package cloudstack-awsapi.x86_64 0:4.1.1-0.el6 will be a downgrade > > ---> Package cloudstack-awsapi.x86_64 0:4.2.0-1.el6 will be erased > > ---> Package cloudstack-cli.x86_64 0:4.1.1-0.el6 will be a downgrade > > ---> Package cloudstack-cli.x86_64 0:4.2.0-1.el6 will be erased > > ---> Package cloudstack-common.x86_64 0:4.1.1-0.el6 will be a downgrade > > ---> Package cloudstack-common.x86_64 0:4.2.0-1.el6 will be erased > > ---> Package cloudstack-management.x86_64 0:4.1.1-0.el6 will be a > downgrade > > ---> Package cloudstack-management.x86_64 0:4.2.0-1.el6 will be erased > > ---> Package cloudstack-usage.x86_64 0:4.1.1-0.el6 will be a downgrade > > ---> Package cloudstack-usage.x86_64 0:4.2.0-1.el6 will be erased > > --> Finished Dependency Resolution > > > > Dependencies Resolved > > > > > ============================================================================================================================================================= > > Package Arch > > Version Repository > > Size > > > ============================================================================================================================================================= > > Downgrading: > > cloudstack-agent x86_64 > > 4.1.1-0.el6 cloudstack > > 37 M > > cloudstack-awsapi x86_64 > > 4.1.1-0.el6 cloudstack > > 56 M > > cloudstack-cli x86_64 > > 4.1.1-0.el6 cloudstack > > 32 k > > cloudstack-common x86_64 > > 4.1.1-0.el6 cloudstack > > 92 M > > cloudstack-management x86_64 > > 4.1.1-0.el6 cloudstack > > 55 M > > cloudstack-usage x86_64 > > 4.1.1-0.el6 cloudstack > > 37 M > > > > Transaction Summary > > > ============================================================================================================================================================= > > Downgrade 6 Package(s) > > > > Total download size: 278 M > > Is this ok [y/N]: y > > Downloading Packages: > > (1/6): cloudstack-agent-4.1.1-0.el6.x86_64.rpm > > | 37 MB > > 00:13 > > (2/6): cloudstack-awsapi-4.1.1-0.el6.x86_64.rpm > > | 56 MB > 01:01 > > (3/6): cloudstack-cli-4.1.1-0.el6.x86_64.rpm > > | 32 kB > > 00:00 > > (4/6): cloudstack-common-4.1.1-0.el6.x86_64.rpm > > | 92 MB > 01:53 > > (5/6): cloudstack-management-4.1.1-0.el6.x86_64.rpm > > | 55 MB > 00:39 > > (6/6): cloudstack-usage-4.1.1-0.el6.x86_64.rpm > > | 37 MB > > 00:12 > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Total > > 1.2 MB/s | 278 MB > 04:00 > > Running rpm_check_debug > > Running Transaction Test > > Transaction Test Succeeded > > Running Transaction > > Installing : cloudstack-common-4.1.1-0.el6.x86_64 > > > > 1/12 > > Installing : cloudstack-management-4.1.1-0.el6.x86_64 > > > > 2/12 > > warning: /etc/cloudstack/management/db.properties created as > > /etc/cloudstack/management/db.properties.rpmnew > > ln: creating symbolic link > > `/usr/share/cloudstack-management/webapps7080/webapps': File exists > > Please download vhd-util from > > http://download.cloud.com.s3.amazonaws.com/tools/vhd-util and put it in > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/ > > Unable to determine ssl settings for server.xml, please run > > cloudstack-setup-management manually > > Unable to determine ssl settings for tomcat.conf, please run > > cloudstack-setup-management manually > > Installing : cloudstack-awsapi-4.1.1-0.el6.x86_64 > > > > 3/12 > > Installing : cloudstack-agent-4.1.1-0.el6.x86_64 > > > > 4/12 > > warning: /etc/cloudstack/agent/agent.properties created as > > /etc/cloudstack/agent/agent.properties.rpmnew > > Installing : cloudstack-cli-4.1.1-0.el6.x86_64 > > > > 5/12 > > Installing : cloudstack-usage-4.1.1-0.el6.x86_64 > > > > 6/12 > > Replacing db.properties with management server db.properties > > Cleanup : cloudstack-awsapi-4.2.0-1.el6.x86_64 > > > > 7/12 > > Cleanup : cloudstack-management-4.2.0-1.el6.x86_64 > > > > 8/12 > > Stopping cloudstack-management:[ OK ] > > Cleanup : cloudstack-agent-4.2.0-1.el6.x86_64 > > > > 9/12 > > Stopping Cloud Agent: > > Cleanup : cloudstack-common-4.2.0-1.el6.x86_64 > > > > 10/12 > > Cleanup : cloudstack-cli-4.2.0-1.el6.x86_64 > > > > 11/12 > > Cleanup : cloudstack-usage-4.2.0-1.el6.x86_64 > > > > 12/12 > > Verifying : cloudstack-usage-4.1.1-0.el6.x86_64 > > > > 1/12 > > Verifying : cloudstack-awsapi-4.1.1-0.el6.x86_64 > > > > 2/12 > > Verifying : cloudstack-management-4.1.1-0.el6.x86_64 > > > > 3/12 > > Verifying : cloudstack-cli-4.1.1-0.el6.x86_64 > > > > 4/12 > > Verifying : cloudstack-common-4.1.1-0.el6.x86_64 > > > > 5/12 > > Verifying : cloudstack-agent-4.1.1-0.el6.x86_64 > > > > 6/12 > > Verifying : cloudstack-management-4.2.0-1.el6.x86_64 > > > > 7/12 > > Verifying : cloudstack-cli-4.2.0-1.el6.x86_64 > > > > 8/12 > > Verifying : cloudstack-common-4.2.0-1.el6.x86_64 > > > > 9/12 > > Verifying : cloudstack-awsapi-4.2.0-1.el6.x86_64 > > > > 10/12 > > Verifying : cloudstack-usage-4.2.0-1.el6.x86_64 > > > > 11/12 > > Verifying : cloudstack-agent-4.2.0-1.el6.x86_64 > > > > 12/12 > > > > Removed: > > cloudstack-agent.x86_64 0:4.2.0-1.el6 > > cloudstack-awsapi.x86_64 0:4.2.0-1.el6 > > cloudstack-cli.x86_64 0:4.2.0-1.el6 > > cloudstack-common.x86_64 0:4.2.0-1.el6 > > cloudstack-management.x86_64 0:4.2.0-1.el6 > > cloudstack-usage.x86_64 0:4.2.0-1.el6 > > > > Installed: > > cloudstack-agent.x86_64 0:4.1.1-0.el6 > > cloudstack-awsapi.x86_64 0:4.1.1-0.el6 > > cloudstack-cli.x86_64 0:4.1.1-0.el6 > > cloudstack-common.x86_64 0:4.1.1-0.el6 > > cloudstack-management.x86_64 0:4.1.1-0.el6 > > cloudstack-usage.x86_64 0:4.1.1-0.el6 > > > > Complete! > > [root@cs-east-dev1 yum.repos.d]# service cloudstack-agent start > > Starting Cloud Agent: > > [root@cs-east-dev1 yum.repos.d]# service cloudstack-management start > > Starting cloudstack-management: [ OK ] > > [root@cs-east-dev1 yum.repos.d] > > {code} > > > > > > *NOTE*: On the other 3 hosts it only needed to downgrade cloudstack-agent > > and cloudstack-common. > > > > > > After the downgrade I received this silly error in the > management-server.log > > > > {code} > > [root@cs-east-dev1 yum.repos.d]# tail -f > > /var/log/cloudstack/management/management-server.log > > 2013-10-15 15:36:20,138 INFO [web.context.ContextLoader] (main:null) > Root > > WebApplicationContext: initialization started > > 2013-10-15 15:36:20,220 INFO [context.support.XmlWebApplicationContext] > > (main:null) Refreshing Root WebApplicationContext: startup date [Tue Oct > 15 > > 15:36:20 EDT 2013]; root of context hierarchy > > 2013-10-15 15:36:20,276 INFO [factory.xml.XmlBeanDefinitionReader] > > (main:null) Loading XML bean definitions from class path resource > > [applicationContext.xml] > > 2013-10-15 15:36:20,713 INFO [factory.xml.XmlBeanDefinitionReader] > > (main:null) Loading XML bean definitions from class path resource > > [nonossComponentContext.xml] > > 2013-10-15 15:36:22,130 INFO > > [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null) > > JSR-330 'javax.inject.Inject' annotation found and supported for > autowiring > > 2013-10-15 15:36:22,162 INFO [context.support.XmlWebApplicationContext] > > (main:null) Bean 'transactionContextBuilder' of type [class > > com.cloud.utils.db.TransactionContextBuilder] is not eligible for getting > > processed by all BeanPostProcessors (for example: not eligible for > > auto-proxying) > > 2013-10-15 15:36:22,167 INFO [context.support.XmlWebApplicationContext] > > (main:null) Bean 'actionEventInterceptor' of type [class > > com.cloud.event.ActionEventInterceptor] is not eligible for getting > > processed by all BeanPostProcessors (for example: not eligible for > > auto-proxying) > > 2013-10-15 15:36:22,212 INFO > [factory.support.DefaultListableBeanFactory] > > (main:null) Pre-instantiating singletons in > > > org.springframework.beans.factory.support.DefaultListableBeanFactory@757df1b6 > : > > defining beans > > > [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,componentContext,transactionContextBuilder,actionEventInterceptor,instantiatePostProcessor,onwireRegistry,messageSerializer,transportProvider,rpcProvider,eventBus,serviceOfferingDaoImpl,diskOfferingDaoImpl,dataCenterDaoImpl,hostPodDaoImpl,vlanDaoImpl,userDaoImpl,VMTemplateDaoImpl,hypervisorCapabilitiesDaoImpl,accountDaoImpl,accountDetailsDaoImpl,accountJoinDaoImpl,accountVlanMapDaoImpl,agentUpgradeDaoImpl,alertDaoImpl,asyncJobDaoImpl,asyncJobJoinDaoImpl,autoScalePolicyConditionMapDaoImpl,autoScalePolicyDaoImpl,autoScaleVmGroupDaoImpl,autoScaleVmGroupPolicyMapDaoImpl,autoScaleVmProfileDaoImpl,capacityDaoImpl,certificateDaoImpl,clusterDaoImpl,clusterDetailsDaoImpl,commandExecLogDaoImpl,conditionDaoImpl,consoleProxyDaoImpl,counterDaoImpl,dataCenterIpAddressDaoImpl,dataCenterJoinDaoImpl,dataCenterLinkLocalIpAddressDaoImpl,dataCenterVnetDaoImpl,dataStoreProviderDaoImpl,dcDetailsDaoImpl,engineDcDetailsDaoImpl,diskOfferingJoinDaoImpl,domainDaoImpl,domainRouterDaoImpl,domainRouterJoinDaoImpl,elasticLbVmMapDaoImpl,engineClusterDaoImpl,engineDataCenterDaoImpl,engineHostDaoImpl,engineHostPodDaoImpl,eventDaoImpl,eventJoinDaoImpl,externalFirewallDeviceDaoImpl,externalLoadBalancerDeviceDaoImpl,externalPublicIpStatisticsDaoImpl,firewallRulesCidrsDaoImpl,firewallRulesDaoImpl,guestOSCategoryDaoImpl,guestOSDaoImpl,highAvailabilityDaoImpl,hostDaoImpl,hostDetailsDaoImpl,engineHostDetailsDaoImpl,hostJoinDaoImpl,engineHostTagsDaoImpl,hostTagsDaoImpl,hostTransferMapDaoImpl,iPAddressDaoImpl,identityDaoImpl,imageDaoStoreDaoImpl,imageDataDaoImpl,imageDataStoreProviderDaoImpl,inlineLoadBalancerNicMapDaoImpl,instanceGroupDaoImpl,instanceGroupJoinDaoImpl,instanceGroupVMMapDaoImpl,itWorkDaoImpl,keystoreDaoImpl,lBStickinessPolicyDaoImpl,launchPermissionDaoImpl,loadBalancerDaoImpl,loadBalancerVMMapDaoImpl,managementServerHostDaoImpl,managementServerHostPeerDaoImpl,networkAccountDaoImpl,networkDaoImpl,networkDomainDaoImpl,networkExternalFirewallDaoImpl,networkExternalLoadBalancerDaoImpl,networkOfferingDaoImpl,networkOfferingServiceMapDaoImpl,networkOpDaoImpl,networkRuleConfigDaoImpl,networkServiceMapDaoImpl,nicDaoImpl,objectInDataStoreDaoImpl,ovsTunnelInterfaceDaoImpl,ovsTunnelNetworkDaoImpl,physicalNetworkDaoImpl,physicalNetworkIsolationMethodDaoImpl,physicalNetworkServiceProviderDaoImpl,physicalNetworkTagDaoImpl,physicalNetworkTrafficTypeDaoImpl,podVlanDaoImpl,podVlanMapDaoImpl,portForwardingRulesDaoImpl,primaryDataStoreDaoImpl,primaryDataStoreDetailsDaoImpl,privateIpDaoImpl,projectAccountDaoImpl,projectAccountJoinDaoImpl,projectDaoImpl,projectInvitationDaoImpl,projectInvitationJoinDaoImpl,projectJoinDaoImpl,regionDaoImpl,remoteAccessVpnDaoImpl,resourceCountDaoImpl,resourceLimitDaoImpl,resourceTagJoinDaoImpl,resourceTagsDaoImpl,routerNetworkDaoImpl,s3DaoImpl,sSHKeyPairDaoImpl,secondaryStorageVmDaoImpl,securityGroupDaoImpl,securityGroupJoinDaoImpl,securityGroupRuleDaoImpl,securityGroupRulesDaoImpl,securityGroupVMMapDaoImpl,securityGroupWorkDaoImpl,serviceOfferingJoinDaoImpl,site2SiteCustomerGatewayDaoImpl,site2SiteVpnConnectionDaoImpl,site2SiteVpnGatewayDaoImpl,snapshotDao2Impl,snapshotDaoImpl,snapshotPolicyDaoImpl,snapshotScheduleDaoImpl,stackMaidDaoImpl,staticRouteDaoImpl,storageNetworkIpAddressDaoImpl,storageNetworkIpRangeDaoImpl,storagePoolDaoImpl,storagePoolDetailsDaoImpl,storagePoolHostDaoImpl,storagePoolJoinDaoImpl,storagePoolWorkDaoImpl,swiftDaoImpl,syncQueueDaoImpl,syncQueueItemDaoImpl,templatePrimaryDataStoreDaoImpl,uploadDaoImpl,usageDaoImpl,usageEventDaoImpl,usageIPAddressDaoImpl,usageJobDaoImpl,usageLoadBalancerPolicyDaoImpl,usageNetworkDaoImpl,usageNetworkOfferingDaoImpl,usagePortForwardingRuleDaoImpl,usageSecurityGroupDaoImpl,usageStorageDaoImpl,usageVMInstanceDaoImpl,usageVPNUserDaoImpl,usageVolumeDaoImpl,userAccountDaoImpl,userAccountJoinDaoImpl,userIpv6AddressDaoImpl,userStatisticsDaoImpl,userStatsLogDaoImpl,userVmDetailsDaoImpl,userVmJoinDaoImpl,vMComputeTagDaoImpl,vMEntityDaoImpl,vMInstanceDaoImpl,vMNetworkMapDaoImpl,vMReservationDaoImpl,vMRootDiskTagDaoImpl,vMTemplateDetailsDaoImpl,vMTemplateHostDaoImpl,vMTemplatePoolDaoImpl,vMTemplateS3DaoImpl,vMTemplateSwiftDaoImpl,vMTemplateZoneDaoImpl,versionDaoImpl,virtualRouterProviderDaoImpl,vmRulesetLogDaoImpl,volumeDao2Impl,volumeDaoImpl,volumeHostDaoImpl,volumeJoinDaoImpl,volumeReservationDaoImpl,vpcDaoImpl,vpcGatewayDaoImpl,vpcOfferingDaoImpl,vpcOfferingServiceMapDaoImpl,vpnUserDaoImpl,userVmDaoImpl,portProfileDaoImpl,clusterVSMMapDaoImpl,encryptionSecretKeyChecker,StaticRoleBasedAPIAccessChecker,databaseIntegrityChecker,domainChecker,basicAgentAuthManager,MD5UserAuthenticator,LDAPUserAuthenticator,PlainTextUserAuthenticator,Ovs,SecurityGroupProvider,VirtualRouter,VpcVirtualRouter,elasticLoadBalancerElement,firstFitAllocator,randomAllocator,FirstFitRouting,LocalStoragePoolAllocator,FirstFitStoragePoolAllocator,UserConcentratedAllocator,useLocalForRootAllocator,garbageCollectingStoragePoolAllocator,randomStoragePoolAllocator,recreatableFencer,recreateHostAllocator,secondaryStorageVmDefaultAllocator,ConsoleProxyAllocator,ExteralIpAddressAllocator,hyervisorTemplateAdapter,clusterAlertAdapter,consoleProxyAlertAdapter,secondaryStorageVmAlertAdapter,clusterServiceServletAdapter,CheckOnAgentInvestigator,XenServerInvestigator,UserVmDomRInvestigator,ManagementIPSystemVMInvestigator,XenServerFencer,KVMFencer,OvmFencer,XcpServerDiscoverer,SecondaryStorageDiscoverer,KvmServerDiscoverer,HypervServerDiscoverer,OvmDiscoverer,dummyHostDiscoverer,UserDispersingPlanner,UserConcentratedPodPlanner,clusterBasedAgentLoadBalancerPlanner,StorageNetworkGuru,ExternalGuestNetworkGuru,PublicNetworkGuru,PodBasedNetworkGuru,ControlNetworkGuru,DirectNetworkGuru,DirectPodBasedNetworkGuru,OvsGuestNetworkGuru,PrivateNetworkGuru,XenServerGuru,KVMGuru,OvmGuru,HypervisorPlannerSelector,accountManagerImpl,managementServerImpl,storageManagerImpl,FirstFitPlanner,resourceManagerImpl,highAvailabilityManagerExtImpl,clusteredVirtualMachineManagerImpl,networkManagerImpl,networkModelImpl,configurationServerImpl,clusterManagerImpl,clusteredAgentManagerImpl,userVmManagerImpl,consoleProxyManagerImpl,securityGroupManagerImpl2,premiumSecondaryStorageManagerImpl,ipv6AddressManagerImpl,apiRateLimitServiceImpl,alertManagerImpl,asyncJobExecutorContextImpl,asyncJobManagerImpl,autoScaleManagerImpl,capacityManagerImpl,clusterFenceManagerImpl,configurationManagerImpl,dataStoreProviderManagerImpl,domainManagerImpl,downloadMonitorImpl,elasticLoadBalancerManagerImpl,entityManagerImpl,externalLoadBalancerUsageManagerImpl,externalNetworkDeviceManagerImpl,firewallManagerImpl,hypervisorGuruManagerImpl,identityServiceImpl,keystoreManagerImpl,loadBalancingRulesManagerImpl,networkACLManagerImpl,networkServiceImpl,networkUsageManagerImpl,oCFS2ManagerImpl,ovsTunnelManagerImpl,projectManagerImpl,queryManagerImpl,regionManagerImpl,regionServiceImpl,remoteAccessVpnManagerImpl,resourceLimitManagerImpl,rulesManagerImpl,s3ManagerImpl,site2SiteVpnManagerImpl,snapshotManagerImpl,snapshotSchedulerImpl,storageNetworkManagerImpl,swiftManagerImpl,syncQueueManagerImpl,taggedResourceManagerImpl,templateManagerImpl,upgradeManagerImpl,uploadMonitorImpl,usageServiceImpl,virtualNetworkApplianceManagerImpl,vpcManagerImpl,vpcVirtualNetworkApplianceManagerImpl,cloudZonesStartupProcessor,managementServerNode,actionEventUtils,agentMonitor,alertGenerator,apiDBUtils,apiDiscoveryServiceImpl,apiDispatcher,apiResponseHelper,apiServer,apiServlet,cloudOrchestrator,clusterRestService,consoleProxyServlet,dataCenterResourceManagerImpl,dataDisk,dataMotionServiceImpl,dataObjectManagerImpl,dataStoreManagerImpl,defaultEndPointSelector,defaultImageDataStoreProvider,defaultImageMotionStrategy,defaultPrimaryDataStoreProviderManagerImpl,defaultPrimaryDatastoreProviderImpl,hypervisorBasedSnapshot,hypervsiorHostEndPointRpcServer,iSCSI,ISO,imageDataFactoryImpl,imageDataManagerImpl,imageDataStoreHelper,imageDataStoreManagerImpl,imageFormatHelper,imageMotionServiceImpl,imageServiceImpl,iso,networkFileSystem,networkRestService,OVA,objectInDataStoreManagerImpl,podRestService,primaryDataStoreHelper,provisioningServiceImpl,QCOW2,registerCompleteServlet,rootDisk,snapshotDataFactoryImpl,snapshotServiceImpl,statsCollector,templateInstallStrategyImpl,unknown,usageEventUtils,userContextInitializer,VHD,vMEntityManagerImpl,virtualMachineEntityImpl,virtualMachineRestService,volumeDataFactoryImpl,volumeManagerImpl,volumeRestService,volumeServiceImpl,volumeTypeHelper,zoneRestService,premiumDatabaseUpgradeChecker,configurationDaoImpl,VmwareFencer,vmwareServerDiscoverer,VMwareGuru,VmwareInvestigator,VmwareManager,vmwareContextFactory,niciraNvpDaoImpl,niciraNvpNicMappingDaoImpl,niciraNvpRouterMappingDaoImpl,NiciraNvpGuestNetworkGuru,NiciraNvp,lunDaoImpl,poolDaoImpl,netappVolumeDaoImpl,NetappManager,JuniperSRX,netScalerPodDaoImpl,Netscaler,F5BigIP,ciscoNexusVSMDeviceDaoImpl,CiscoNexus1000vVSM,bigSwitchVnsDaoImpl,bigSwitchVnsElement,userAuthenticators,securityCheckers,resourceDiscoverers,haInvestigators,haFenceBuilders,deploymentPlanners,podAllocators,hostAllocators,storagePoolAllocators,ipDeployers,dhcpProviders,networkGurus,networkElements,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; > > root of factory hierarchy > > 2013-10-15 15:36:22,219 INFO [utils.component.ComponentContext] > > (main:null) Setup Spring Application context > > 2013-10-15 15:36:22,515 DEBUG [utils.crypt.EncryptionSecretKeyChecker] > > (main:null) Encryption Type: file > > 2013-10-15 15:36:27,126 INFO [cloud.serializer.GsonHelper] (main:null) > > Default Builder inited. > > 2013-10-15 15:36:27,901 INFO [hypervisor.vmware.VmwareServerDiscoverer] > > (main:null) VmwareServerDiscoverer is constructed > > 2013-10-15 15:36:28,117 INFO [web.context.ContextLoader] (main:null) > Root > > WebApplicationContext: initialization completed in 7978 ms > > 2013-10-15 15:36:28,134 INFO [cloud.utils.LogUtils] (main:null) log4j > > configuration found at /etc/cloudstack/management/log4j-cloud.xml > > 2013-10-15 15:36:28,151 INFO > > [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null) > > JSR-330 'javax.inject.Inject' annotation found and supported for > autowiring > > 2013-10-15 15:36:28,152 INFO > > [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null) > > JSR-330 'javax.inject.Inject' annotation found and supported for > autowiring > > 2013-10-15 15:36:28,153 INFO > > [factory.annotation.AutowiredAnnotationBeanPostProcessor] (main:null) > > JSR-330 'javax.inject.Inject' annotation found and supported for > autowiring > > 2013-10-15 15:36:33,514 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.CloudStackAccountDaoImpl_EnhancerByCloudStack_3de75b6e > > 2013-10-15 15:36:33,516 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.OfferingDaoImpl_EnhancerByCloudStack_d2f1cac8 > > 2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.persist.dao.SMetaDaoImpl_EnhancerByCloudStack_ef98bac0 > > 2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.UserCredentialsDaoImpl_EnhancerByCloudStack_909a85df > > 2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.MultipartMetaDaoImpl_EnhancerByCloudStack_d7013d7f > > 2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.CloudStackConfigurationDaoImpl_EnhancerByCloudStack_fecccd9d > > 2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.BucketPolicyDaoImpl_EnhancerByCloudStack_56673762 > > 2013-10-15 15:36:33,517 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.SObjectItemDaoImpl_EnhancerByCloudStack_3a79ef2b > > 2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.MHostMountDaoImpl_EnhancerByCloudStack_8f7d233a > > 2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.persist.dao.MHostDaoImpl_EnhancerByCloudStack_165457b > > 2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.MultiPartUploadsDaoImpl_EnhancerByCloudStack_bcd9eaa8 > > 2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.persist.dao.SBucketDaoImpl_EnhancerByCloudStack_8e162773 > > 2013-10-15 15:36:33,518 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.persist.dao.SAclDaoImpl_EnhancerByCloudStack_104ffef > > 2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.MultiPartPartsDaoImpl_EnhancerByCloudStack_88d4152a > > 2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.CloudStackSvcOfferingDaoImpl_EnhancerByCloudStack_882a018d > > 2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_29f7bbc5 > > 2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStack_c20b110 > > 2013-10-15 15:36:33,519 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_10d22bb6 > > 2013-10-15 15:36:33,520 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_635f1066 > > 2013-10-15 15:36:33,561 INFO [utils.component.ComponentContext] > > (Timer-1:null) Running SystemIntegrityChecker encryptionSecretKeyChecker > > 2013-10-15 15:36:33,563 DEBUG [utils.crypt.EncryptionSecretKeyChecker] > > (Timer-1:null) Encryption Type: file > > 2013-10-15 15:36:33,563 WARN [utils.crypt.EncryptionSecretKeyChecker] > > (Timer-1:null) Encryption already enabled, is check() called twice? > > 2013-10-15 15:36:33,564 INFO [utils.component.ComponentContext] > > (Timer-1:null) Running SystemIntegrityChecker databaseIntegrityChecker > > 2013-10-15 15:36:33,564 INFO [cloud.upgrade.DatabaseIntegrityChecker] > > (Timer-1:null) Grabbing lock to check for database integrity. > > 2013-10-15 15:36:33,750 INFO [cloud.upgrade.DatabaseIntegrityChecker] > > (Timer-1:null) Performing database integrity check > > 2013-10-15 15:36:33,754 INFO [utils.component.ComponentContext] > > (main:null) Configuring > > > com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloudStack_8e8faadb > > 2013-10-15 15:36:33,763 DEBUG [cloud.upgrade.DatabaseIntegrityChecker] > > (Timer-1:null) No duplicate hosts with the same local storage found in > > database > > 2013-10-15 15:36:33,767 DEBUG [upgrade.dao.VersionDaoImpl] (Timer-1:null) > > Checking to see if the database is at a version before it was the version > > table is created > > 2013-10-15 15:36:33,787 INFO [utils.component.ComponentContext] > > (Timer-1:null) Running SystemIntegrityChecker managementServerNode > > 2013-10-15 15:36:33,788 ERROR [utils.component.ComponentContext] > > (Timer-1:null) *System integrity check failed. Refuse to startup* > > [root@cs-east-dev1 yum.repos.d]# > > {code} > > > > > > I googled this issue and found the following discussion: > > > > > http://mail-archives.apache.org/mod_mbox/cloudstack-issues/201309.mbox/%3CJIRA.12671428.1380572215050.5987.1380572367958@arcas%3E > > > > > > and this discussion points to the following defects: > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-4770 && > > https://issues.apache.org/jira/browse/CLOUDSTACK-4170 > > > > > > But defect CLOUDSTACK-4770 is set to *minor *and there doesn't seem to be > > any activity on it. > > > > In the CLOUDSTACK-4770 defect report Richard Chatterton mentions having a > > MAC address of 00:00:00:00:00:00, so I performed the same steps as > Richard > > (except for version 4.1.1) and got the following output: > > > > {code} > > [root@cs-east-dev1 ~]# java -classpath > > > /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/cloud-utils-4.1.1.jar > > com.cloud.utils.net.MacAddress > > *addr in integer is 0* > > *addr in bytes is 0 0 0 0 0 0* > > *addr in char is 00:00:00:00:00:00* > > [root@cs-east-dev1 ~]# ifconfig -a > > *cloud0 Link encap:Ethernet HWaddr 00:00:00:00:00:00* > > inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 > > inet6 addr: fe80::9c89:feff:feca:45e5/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:1752 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:0 (0.0 b) TX bytes:73800 (72.0 KiB) > > > > cloudbr0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A > > inet addr:10.97.36.115 Bcast:10.97.36.255 Mask:255.255.255.0 > > inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:36186 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:3830135 (3.6 MiB) TX bytes:746 (746.0 b) > > > > eth0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A > > inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:44652 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:251 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:5815431 (5.5 MiB) TX bytes:19286 (18.8 KiB) > > Interrupt:17 > > > > eth1 Link encap:Ethernet HWaddr 00:18:F8:0B:53:2B > > inet addr:10.97.36.105 Bcast:10.97.36.255 Mask:255.255.255.0 > > inet6 addr: fe80::218:f8ff:fe0b:532b/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:542135 errors:0 dropped:27 overruns:0 frame:0 > > TX packets:422950 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:506474573 (483.0 MiB) TX bytes:157977312 (150.6 MiB) > > > > lo Link encap:Local Loopback > > inet addr:127.0.0.1 Mask:255.0.0.0 > > inet6 addr: ::1/128 Scope:Host > > UP LOOPBACK RUNNING MTU:16436 Metric:1 > > RX packets:547879 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:547879 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:122259530 (116.5 MiB) TX bytes:122259530 (116.5 MiB) > > > > virbr0 Link encap:Ethernet HWaddr 52:54:00:40:5C:17 > > inet addr:192.168.122.1 Bcast:192.168.122.255 > Mask:255.255.255.0 > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > > > virbr0-nic Link encap:Ethernet HWaddr 52:54:00:40:5C:17 > > BROADCAST MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:500 > > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > > > [root@cs-east-dev1 ~]# > > {code} > > > > > > So... then I was curious what would happen if I simply restarted the > > network service: > > > > {code} > > [root@cs-east-dev1 ~]# service network status > > Configured devices: > > lo cloudbr0 eth0 eth1 > > Currently active devices: > > lo eth0 eth1 cloudbr0 virbr0 cloud0 > > [root@cs-east-dev1 ~]# service network restart > > Shutting down interface cloudbr0: [ OK ] > > Shutting down interface eth0: [ OK ] > > Shutting down interface eth1: [ OK ] > > Shutting down loopback interface: [ OK ] > > Bringing up loopback interface: [ OK ] > > Bringing up interface eth0: [ OK ] > > Bringing up interface eth1: > > Determining IP information for eth1... done. > > [ OK ] > > Bringing up interface cloudbr0: [ OK ] > > [root@cs-east-dev1 ~]# ifconfig -a > > cloudbr0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A > > inet addr:10.97.36.115 Bcast:10.97.36.255 Mask:255.255.255.0 > > inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:30 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:9 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:2639 (2.5 KiB) TX bytes:566 (566.0 b) > > > > cloud0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 > > inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 > > inet6 addr: fe80::9c89:feff:feca:45e5/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:1752 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:0 (0.0 b) TX bytes:73800 (72.0 KiB) > > > > eth0 Link encap:Ethernet HWaddr 18:A9:05:B7:44:7A > > inet6 addr: fe80::1aa9:5ff:feb7:447a/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:65 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:15 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:7266 (7.0 KiB) TX bytes:1132 (1.1 KiB) > > Interrupt:17 > > > > eth1 Link encap:Ethernet HWaddr 00:18:F8:0B:53:2B > > inet addr:10.97.36.105 Bcast:10.97.36.255 Mask:255.255.255.0 > > inet6 addr: fe80::218:f8ff:fe0b:532b/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:546735 errors:0 dropped:27 overruns:0 frame:0 > > TX packets:428375 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:506928715 (483.4 MiB) TX bytes:160011479 (152.5 MiB) > > > > lo Link encap:Local Loopback > > inet addr:127.0.0.1 Mask:255.0.0.0 > > inet6 addr: ::1/128 Scope:Host > > UP LOOPBACK RUNNING MTU:16436 Metric:1 > > RX packets:548153 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:548153 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:122274810 (116.6 MiB) TX bytes:122274810 (116.6 MiB) > > > > virbr0 Link encap:Ethernet HWaddr 52:54:00:40:5C:17 > > inet addr:192.168.122.1 Bcast:192.168.122.255 > Mask:255.255.255.0 > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > > > virbr0-nic Link encap:Ethernet HWaddr 52:54:00:40:5C:17 > > BROADCAST MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:500 > > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > > > [root@cs-east-dev1 ~]# > > {code} > > > > > > > > Notice how now cloudbr0 is listed first, and that has a valid MAC > address. > > Believe it or not, now the management server starts up fine, however... > the > > System VMs still wouldn't start. They were stuck in the "Starting" state. > > So... I downloaded the System ISO seed again > > > > {code} > > [root@cs-east-dev1 ~]# > > > /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt > > -m /export/secondary -u > > > http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2-h > > kvm -F > > --2013-10-15 16:31:45-- > > > http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2 > > Resolving download.cloud.com... 207.171.187.118 > > Connecting to download.cloud.com|207.171.187.118|:80... connected. > > HTTP request sent, awaiting response... 200 OK > > Length: 286036668 (273M) [binary/octet-stream] > > Saving to: > > > â/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2â > > > > > 100%[===================================================================================================================>] > > 286,036,668 4.15M/s in 73s > > > > 2013-10-15 16:32:59 (3.73 MB/s) - > > > â/usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2â > > saved [286036668/286036668] > > > > Uncompressing to > > > /usr/share/cloudstack-common/scripts/storage/secondary/1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2.tmp > > (type bz2)...could take a long time > > Moving to > > > /export/secondary/template/tmpl/1/3///1d06b934-e947-4a29-8dbc-6901d3de27a6.qcow2...could > > take a while > > Successfully installed system VM template to > > /export/secondary/template/tmpl/1/3/ > > [root@cs-east-dev1 ~]# > > {code} > > > > > > Once that was complete I destroyed the system VMs and restarted > > cloudstack-management. Now everything seems back to normal. > > > > Bottom line, I don't feel that 4.2 is production ready yet for KVM on > > CentOS 6.4. I hope this helps someone else... > > > > -- > > > > Best Regards, > > > > > > > > Adam Scarcella > > > > Sr. Support Engineer | Spotfire, TIBCO Software Inc. > > > > Web: http://spotfire.tibco.com/support > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
