Re: DNS resolution with advanced networks between VLAN's

[Yuri Kogun]

Thanks, ErikWill check it out. 
Best regards Yuri

Sent from Outlook

_
From: Erik Weber 
Sent: Friday, May 15, 2015 6:35 pm
Subject: Re: DNS resolution with advanced networks between VLAN's
To:  


   The globodns plugin could possibly solve this by serving all 
domains.   

 Erik   

 Den fredag 15. mai 2015 skrev Yuri Kogun  følgende:   

 > Hi,   
 > I wonder if anybody had a similar problem. Our cloudstack zone is setup   
 > with advanced networking and we have setup a couple  VLAN's. Each VLAN has   
 > it's own network mask and IP range. The host name is assigned via Dhclient   
 > hook and derived from the VM name when vm is started. The DNS resolution   
 > works fine inside the VLAN but it fails for the inter VLAN traffic. Is   
 > there a way to address this ?   
 >   
 > We use different VLAN's to keep VM's with static and dynamic IP's apart to   
 > avoid IP conflicts.   
 >   
 > Best regards   
 > Yuri Kogun

DNS resolution with advanced networks between VLAN's

[Yuri Kogun]

Hi, 
I wonder if anybody had a similar problem. Our cloudstack zone is setup with 
advanced networking and we have setup a couple  VLAN's. Each VLAN has it's own 
network mask and IP range. The host name is assigned via Dhclient hook and 
derived from the VM name when vm is started. The DNS resolution works fine 
inside the VLAN but it fails for the inter VLAN traffic. Is there a way to 
address this ? 

We use different VLAN's to keep VM's with static and dynamic IP's apart to 
avoid IP conflicts.

Best regards 
Yuri Kogun    

RE: Xen - Deploying servers from a template

[Yuri Kogun]

Yes, it works fine with the shared storage. This is local storage on Xen server 
issue.  I'm happy to provide more info if needed 

Best regards 
Yuri 


> From: geoff.higginbot...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: Re: Xen - Deploying servers from a template
> Date: Wed, 29 Apr 2015 13:05:59 +
>
> Hi Yuri,
>
> Thanks for the update. Would I be right in saying this is a local storage 
> issue and if using shared storage the prepare template api functions as 
> expected.
>
> I think I've only tested when using shared storage so I will get our 
> developers to look at this to see if we can include the fix in 4.5.1
>
> Regards
>
> Geoff Higginbottom
> CTO / Cloud Architect
>
> D: +44 20 3603 0542 | S: +44 20 3603 
> 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com> | 
> www.shapeblue.com | 
> Twitter:@cloudstackguru<https://twitter.com/#!/cloudstackguru>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 
> 4HS
>
>
> On 29 Apr 2015, at 15:45, Yuri Kogun 
> mailto:yko...@outlook.com>> wrote:
>
> Hi Geoff,
>
> We have raised the issue with citrix cloudPlatform and I believe they are 
> working on it and should contribute the fix back to cloudstack.
>
> The problem itself as follows
>
> if you have more than one host in the XEN cluster with the local storage and 
> send the prepare_template api call for a new template the secondary storage 
> template NFS directory will be mounted as SR on all hosts in the cluster and 
> you will get the following error in the log
>
> GetStorageStats Exception:SR_BACKEND_FAILURE_40The SR scan failed 
> [opterr=['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", 
> "uuid", "9fc07770-b9c0-4db6-a0e0-48a24c901db7")']]
>
> The template will be copied to the local storage on only server in the 
> cluster and you will have to repeat the command to copy the template again. 
> The only way to check if the template was copied to the local storage will be 
> to query the cloud.template_spool_ref table.
>
> Best regards
> Yuri
>
> 
> From: 
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com>
> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> Subject: Re: Xen - Deploying servers from a template
> Date: Wed, 29 Apr 2015 10:28:29 +
>
> Hi Yuri,
>
> Thanks for adding some more detail to the thread.
>
> You say that the prepare template api does not work for you, I have not had 
> the same experience so could you share more details and ideally log a jira 
> ticket etc
>
> Regards
>
> Geoff Higginbottom
> CTO / Cloud Architect
>
> D: +44 20 3603 0542 | S: +44 20 3603 
> 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com><mailto:geoff.higginbot...@shapeblue.com>
>  | www.shapeblue.com<http://www.shapeblue.com> | 
> Twitter:@cloudstackguru<https://twitter.com/#!/cloudstackguru>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 
> 4HS
>
>
> On 29 Apr 2015, at 14:10, Yuri Kogun 
> mailto:yko...@outlook.com><mailto:yko...@outlook.com>> 
> wrote:
>
> Just wanted to add from the personal experience that the process of copying 
> template is sequential as the secondary storage can be mounted only on one 
> host in the cluster (due to SR locking issue) . We use local storage in 
> production and normally prepare the template by starting the vm with is on 
> every host in the cluster in the quiet time. The prepare_template API call 
> does not work properly.
> The other problem is that the XEN server host is unusable until the template 
> copy is finished and all other calls to start VM's on the host will have to 
> wait.
>
> Best regards
> Yuri Kogun
> 
> From: 
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com><mailto:geoff.higginbot...@shapeblue.com>
> To: 
> users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>
> Subject: Re: Xen - Deploying servers from a template
> Date: Tue, 28 Apr 2015 11:58:55 +
>
> The first time a template is used, it is copied from Secondary to Primary,
> then the VM is created, and in the case of XenServer is created as a
> Œlinked clone¹. This linking of the VM to the Template means that whilst
> there are VMs allocated to Primary Storage, the templates that the VMs are

RE: Xen - Deploying servers from a template

[Yuri Kogun]

Hi Geoff, 

We have raised the issue with citrix cloudPlatform and I believe they are 
working on it and should  contribute the fix back to cloudstack. 

The problem itself as follows 

if you have more than one host in the XEN cluster with the local storage and 
send the prepare_template api call for a new template the secondary storage 
template NFS directory will be mounted as SR on all hosts in the cluster and 
you will get the following error in the log 

GetStorageStats Exception:SR_BACKEND_FAILURE_40The SR scan failed  
[opterr=['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", 
"uuid", "9fc07770-b9c0-4db6-a0e0-48a24c901db7")']]

The template will be copied to the local storage on only  server in the cluster 
and you will have to repeat the command to copy the template again. The only 
way to check if the template was copied to the local storage will be to query 
the  cloud.template_spool_ref table. 

Best regards 
Yuri 


> From: geoff.higginbot...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: Re: Xen - Deploying servers from a template
> Date: Wed, 29 Apr 2015 10:28:29 +
>
> Hi Yuri,
>
> Thanks for adding some more detail to the thread.
>
> You say that the prepare template api does not work for you, I have not had 
> the same experience so could you share more details and ideally log a jira 
> ticket etc
>
> Regards
>
> Geoff Higginbottom
> CTO / Cloud Architect
>
> D: +44 20 3603 0542 | S: +44 20 3603 
> 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com> | 
> www.shapeblue.com | 
> Twitter:@cloudstackguru<https://twitter.com/#!/cloudstackguru>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 
> 4HS
>
>
> On 29 Apr 2015, at 14:10, Yuri Kogun 
> mailto:yko...@outlook.com>> wrote:
>
> Just wanted to add from the personal experience that the process of copying 
> template is sequential as the secondary storage can be mounted only on one 
> host in the cluster (due to SR locking issue) . We use local storage in 
> production and normally prepare the template by starting the vm with is on 
> every host in the cluster in the quiet time. The prepare_template API call 
> does not work properly.
> The other problem is that the XEN server host is unusable until the template 
> copy is finished and all other calls to start VM's on the host will have to 
> wait.
>
> Best regards
> Yuri Kogun
> 
> From: 
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com>
> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> Subject: Re: Xen - Deploying servers from a template
> Date: Tue, 28 Apr 2015 11:58:55 +
>
> The first time a template is used, it is copied from Secondary to Primary,
> then the VM is created, and in the case of XenServer is created as a
> Œlinked clone¹. This linking of the VM to the Template means that whilst
> there are VMs allocated to Primary Storage, the templates that the VMs are
> using will always remain on Primary Storage.
>
> If a 2nd VM is created based on the same template, the template is already
> on Primary Storage so is not copied again.
>
> If all the VMs which were using a Template are deleted or migrated to
> alternate storage, the Template is no longer required and will be deleted
> from Primary Storage.
>
> This automatic clean up can be disabled by updating the global setting
> storage.template.cleanup.enabled
>
> Regards
>
> Geoff Higginbottom
> CTO / Cloud Architect
>
> D: +44 20 3603 0542  | S: +44 20 3603 0540
>  | M: +447968161581 
>
> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com> | 
> www.shapeblue.com<http://www.shapeblue.com>
>  | Twitter:@cloudstackguru
> <https://twitter.com/#!/cloudstackguru>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
> 
>
>
>
>
> On 28/04/2015 15:13, "cs user" 
> mailto:acldstk...@gmail.com>> wrote:
>
> Hi All,
>
> Quick question with regards to xenserver and deploying a cloudstack
> instance from a template stored within secondary storage.
>
> In this case it is secondary storage using NFS and deploying to local disk
> on the xenserver. (cloudstack 4.3, Xen 6.0.1)
>
> I believe this script is used to pull down the template:
> /opt/cloud/bin/copy_vhd_from_secondarystorage.sh
>
> I'm seeing a big variation when deploying 3 servers, for example. 2 of the
> servers might start very quickly and one may take longer.
>
> Is there any caching of th

RE: Xen - Deploying servers from a template

[Yuri Kogun]

Just wanted to add  from the personal experience that the process of copying 
template is sequential as the secondary storage can be mounted only on one host 
in the cluster (due to SR  locking issue)  .  We use local storage in 
production and normally prepare the template by starting the vm with is on 
every host in the cluster in the quiet time. The prepare_template API call does 
not work properly.
The other problem is that the XEN server host is unusable until the template 
copy is finished and all other calls to start VM's on the host will have to 
wait. 

Best regards 
Yuri Kogun 

> From: geoff.higginbot...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: Re: Xen - Deploying servers from a template
> Date: Tue, 28 Apr 2015 11:58:55 +
>
> The first time a template is used, it is copied from Secondary to Primary,
> then the VM is created, and in the case of XenServer is created as a
> Œlinked clone¹. This linking of the VM to the Template means that whilst
> there are VMs allocated to Primary Storage, the templates that the VMs are
> using will always remain on Primary Storage.
>
> If a 2nd VM is created based on the same template, the template is already
> on Primary Storage so is not copied again.
>
> If all the VMs which were using a Template are deleted or migrated to
> alternate storage, the Template is no longer required and will be deleted
> from Primary Storage.
>
> This automatic clean up can be disabled by updating the global setting
> storage.template.cleanup.enabled
>
> Regards
>
> Geoff Higginbottom
> CTO / Cloud Architect
>
> D: +44 20 3603 0542  | S: +44 20 3603 0540
>  | M: +447968161581 
>
> geoff.higginbot...@shapeblue.com | www.shapeblue.com
>  | Twitter:@cloudstackguru
> <https://twitter.com/#!/cloudstackguru>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
> 
>
>
>
>
> On 28/04/2015 15:13, "cs user"  wrote:
>
>>Hi All,
>>
>>Quick question with regards to xenserver and deploying a cloudstack
>>instance from a template stored within secondary storage.
>>
>>In this case it is secondary storage using NFS and deploying to local disk
>>on the xenserver. (cloudstack 4.3, Xen 6.0.1)
>>
>>I believe this script is used to pull down the template:
>>/opt/cloud/bin/copy_vhd_from_secondarystorage.sh
>>
>>I'm seeing a big variation when deploying 3 servers, for example. 2 of the
>>servers might start very quickly and one may take longer.
>>
>>Is there any caching of the template taking place on the xen host its
>>self?
>>If a host has previously deployed a template, and is then asked to deploy
>>a
>>new instance using the template, will it have to go back to secondary
>>storage again or can it use a local cache of the vhd file?
>>
>>Many thanks in advance for any responses.
>>
>>Cheers!
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software 
> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure 
> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training 
> Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended 
> solely for the use of the individual to whom it is addressed. Any views or 
> opinions expressed are solely those of the author and do not necessarily 
> represent those of Shape Blue Ltd or related companies. If you are not the 
> intended recipient of this email, you must neither take any action based upon 
> its contents, nor copy or show it to anyone. Please contact the sender if you 
> believe you have received this email in error. Shape Blue Ltd is a company 
> incorporated in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue Ltd. 
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
> operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company 
> registered by The Republic of South Africa and is traded under license from 
> Shape Blue Ltd. ShapeBlue is a registered trademark.
  

RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer

[Yuri Kogun]

Thank you, Somesh 
Will try to upgrade.

Best regards 
Yuri 


> From: somesh.na...@citrix.com
> To: users@cloudstack.apache.org
> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error 
> on XenServer
> Date: Mon, 26 Jan 2015 18:49:54 +
>
>
> Yes, XS will have the correct view since it is the one that's managing the 
> memory. The problem is the way CS calculates the memory overhead. I don't 
> think there is much we can do in terms of configuration.
>
> This is how CS arrives at total ram:
> ram = (long)((ram - dom0Ram - _xsMemoryUsed) * _xsVirtualizationFactor);
> In 4.3.x we derive dom0Ram from dom0's memory-dynamic-max, [dom0Ram = 
> vm.getMemoryStaticMax(conn);].
> The fix was to derive it from dom0's memory-static-max, [dom0Ram = 
> vm.getMemoryStaticMax(conn);].
>
> I see the fix in 4.4.x. So you may want to upgrade to that version to fix 
> this issue.
>
> Regards,
> Somesh
>
>
> -Original Message-
> From: Yuri Kogun [mailto:yko...@outlook.com]
> Sent: Monday, January 26, 2015 11:10 AM
> To: users@cloudstack.apache.org
> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error 
> on XenServer
>
> Hi Somesh,
> Thank you for the reply, yes I found the issue in the Jira.
>
> The affected host was running about 31 VM's and we don't use memory over 
> provisioning at all. Unfortunately cloudstack fails to restart he VM and it 
> is a big issue for us as the restart is a part of a nightly job and we pick 
> it up only in the morning. The only way for us address the issue is to remove 
> 'active' tag from the host in question to stop it from being used for 
> starting new Vm's.
> I also noticed that the available memory is reported correctly in the xen 
> server console, but not inside cloudstack in the management log.
>
>
> Best regards
> Yuri
>
> 
>> From: somesh.na...@citrix.com
>> To: users@cloudstack.apache.org
>> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error 
>> on XenServer
>> Date: Mon, 26 Jan 2015 15:56:01 +
>>
>> Yes, seen this quite a few times. I believe you already found 
>> CLOUDSTACK-2344.
>>
>> Basically, the issue happens when cloudstack's view of available memory is 
>> out of sync (more) than that of XS. This could happen due to incorrect 
>> calculation of memory overhead and Dom0 memory. It is also possible that 
>> your memory overprovisioning value is set too high so please verify that. 
>> You might also want to check how many VMs were running on that particular 
>> host when this error is thrown. When a host is running too many VMs (excess 
>> of 60) then there is a possibility for such issues.
>>
>> Having said that, I believe there are subsequent attempts by cloudstack to 
>> start the VM on other hosts and the VM eventually starts. If not then we may 
>> be looking at a potential defect.
>>
>> Regards,
>> Somesh
>>
>>
>> -Original Message-
>> From: Yuri Kogun [mailto:yko...@outlook.com]
>> Sent: Monday, January 26, 2015 10:26 AM
>> To: users@cloudstack.apache.org
>> Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on 
>> XenServer
>>
>> Hi,
>> I wonder if somebody experienced a similar issue. We have a very busy dev 
>> cloudstack installation with 10 hosts, 3 clusters and more than 300 user 
>> VM's running across the clusters. The cpu over-provisioning is set to 3. 
>> >From time to time we are getting the following an error when starting the 
>> VM.
>>
>> 2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase] 
>> (DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid: 
>> ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
>> nameLabel: Async.VM.start_on
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Jan 26 01:45:24 GMT 2015
>> finished: Mon Jan 26 01:45:24 GMT 2015
>> status: failure
>> residentOn: com.xensource.xenapi.Host@fcaebca8
>> progress: 1.0
>> type: 
>> result:
>> errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>>
>>
>> I traced the job executor for the VM and it looks like the process have 
>> reported that the host have enough RAM 2060627968 to start the VM, which 
>> requestes 1572864000 but for some reason the command failed on the Xen 
>> server.
>

RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer

[Yuri Kogun]

Hi Somesh, 
Thank you for the reply, yes I found the issue in the Jira. 

The affected host was running about 31 VM's and we don't use memory  over 
provisioning at all.  Unfortunately cloudstack fails to restart he VM and it is 
a big issue for us as the restart is a part of a nightly job and we pick it up 
only in the morning.  The only way for us  address the issue is to remove 
'active' tag from the host in question to stop it from being used for starting 
new Vm's. 
I also noticed that the available memory is reported correctly in the xen 
server console, but not inside cloudstack in the management log. 


Best regards 
Yuri 


> From: somesh.na...@citrix.com
> To: users@cloudstack.apache.org
> Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error 
> on XenServer
> Date: Mon, 26 Jan 2015 15:56:01 +
>
> Yes, seen this quite a few times. I believe you already found CLOUDSTACK-2344.
>
> Basically, the issue happens when cloudstack's view of available memory is 
> out of sync (more) than that of XS. This could happen due to incorrect 
> calculation of memory overhead and Dom0 memory. It is also possible that your 
> memory overprovisioning value is set too high so please verify that. You 
> might also want to check how many VMs were running on that particular host 
> when this error is thrown. When a host is running too many VMs (excess of 60) 
> then there is a possibility for such issues.
>
> Having said that, I believe there are subsequent attempts by cloudstack to 
> start the VM on other hosts and the VM eventually starts. If not then we may 
> be looking at a potential defect.
>
> Regards,
> Somesh
>
>
> -Original Message-
> From: Yuri Kogun [mailto:yko...@outlook.com]
> Sent: Monday, January 26, 2015 10:26 AM
> To: users@cloudstack.apache.org
> Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on 
> XenServer
>
> Hi,
> I wonder if somebody experienced a similar issue. We have a very busy dev 
> cloudstack installation with 10 hosts, 3 clusters and more than 300 user VM's 
> running across the clusters. The cpu over-provisioning is set to 3. From time 
> to time we are getting the following an error when starting the VM.
>
> 2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase] 
> (DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid: 
> ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
> nameLabel: Async.VM.start_on
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Mon Jan 26 01:45:24 GMT 2015
> finished: Mon Jan 26 01:45:24 GMT 2015
> status: failure
> residentOn: com.xensource.xenapi.Host@fcaebca8
> progress: 1.0
> type: 
> result:
> errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
>
>
> I traced the job executor for the VM and it looks like the process have 
> reported that the host have enough RAM 2060627968 to start the VM, which 
> requestes 1572864000 but for some reason the command failed on the Xen server.
>
> 2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) 
> Deployment found - P0=VM[User|i-3-160816-VM], 
> P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>  : 
> Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT-->Pool(30))]
> 2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM 
> state transitted from :Starting to Starting with event: OperationRetryvm's 
> original host id: null new host id: 35 host id before state transition: null
> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Hosts's 
> actual total CPU: 44688 and CPU after applying overprovisioning: 58094
> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) We are 
> allocating VM, increasing the used capacity of this host:35
> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current 
> Used CPU: 52500 , Free CPU:5594 ,Requested CPU: 1500
> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current 
> Used RAM: 61220061184 , Free RAM:2060627968 ,Requested RAM: 1572864000
> 2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-8:ctx-c47b7a2f jo

Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer

[Yuri Kogun]

Hi,
I wonder if somebody experienced a similar issue. We have a very busy dev 
cloudstack installation  with 10 hosts, 3 clusters  and more than 300 user VM's 
running across the clusters. The cpu over-provisioning is set to 3. From time 
to time we are getting the following an error when starting the VM.
 
2015-01-26 01:45:25,547 WARN  [c.c.h.x.r.CitrixResourceBase] 
(DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid: 
ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
   nameLabel: Async.VM.start_on
 nameDescription:
   allowedOperations: []
   currentOperations: {}
 created: Mon Jan 26 01:45:24 GMT 2015
    finished: Mon Jan 26 01:45:24 GMT 2015
  status: failure
  residentOn: com.xensource.xenapi.Host@fcaebca8
    progress: 1.0
    type: 
  result:
   errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
 otherConfig: {}
   subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []


I traced the job executor for the VM and it looks like the process have 
reported that the host have enough RAM 2060627968 to start the VM, which 
requestes  1572864000 but for some reason the command failed on the Xen server. 

2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) 
Deployment found  - P0=VM[User|i-3-160816-VM], 
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
 : 
Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT-->Pool(30))]
2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM state 
transitted from :Starting to Starting with event: OperationRetryvm's original 
host id: null new host id: 35 host id before state transition: null
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Hosts's 
actual total CPU: 44688 and CPU after applying overprovisioning: 58094
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) We are 
allocating VM, increasing the used capacity of this host:35
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current 
Used CPU: 52500 , Free CPU:5594 ,Requested CPU: 1500
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current 
Used RAM: 61220061184 , Free RAM:2060627968 ,Requested RAM: 1572864000
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) CPU STATS 
after allocation: for host: 35, old used: 52500, old reserved: 0, actual total: 
44688, total with o
verprovisioning: 58094; new used:54000, reserved:0; requested 
cpu:1500,alloc_from_last:false
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) RAM STATS 
after allocation: for host: 35, old used: 61220061184, old reserved: 0, total: 
63280689152; new use
d: 62792925184, reserved: 0; requested mem: 1572864000,alloc_from_last:false


Please let me know if somebody had a similar problem and managed to fix it. We 
are running xenserver6.2 with cloudstack 4.3.0.1

Bast regards 
Yuri  

RE: Xenserver pool HA in Cloudstack

[Yuri Kogun]

Thank you,  Geoff, 
I will try this later this week.  


> From: geoff.higginbot...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
> Date: Wed, 14 Jan 2015 15:14:04 +
>
> Hi Yuri,
>
> To be totally honest, I've not tested upgrading XenServer once the Pool HA 
> has been enabled, but I would imagine you first disable the Pool HA, then 
> proceed with the upgrade, then re-enable HA at the end.
>
> I would still set manage.xenserver.pool.master=false whilst upgrading the 
> Pool Master.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com
>
> -Original Message-
> From: Yuri Kogun [mailto:yko...@outlook.com]
> Sent: 14 January 2015 11:33
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> Hi Geoff
>
> If the pool master is managed outside the cloudstack, does it mean that there 
> is no need to set manage.xenserver.pool.master=false when applying patched to 
> the Xen cluster with pool HA enabled ?
>
> Will it just be enough to unmanage the cluster in cloudstack , disable the 
> pool HA on the cluster and proceed with the hotfixes or should I follow the 
> procedure described in the documentation?
>
> Thank you
> Yuri
>
> 
>> From: geoff.higginbot...@shapeblue.com
>> To: users@cloudstack.apache.org
>> Subject: RE: Xenserver pool HA in Cloudstack
>> Date: Wed, 7 Jan 2015 17:59:53 +
>>
>> Hi Somesh,
>>
>> I will be addeing appropriate info from this blog into the official 
>> CloudStack Install Docs.
>>
>> Regards
>>
>> Geoff Higginbottom
>>
>> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>>
>> geoff.higginbot...@shapeblue.com
>>
>> -Original Message-
>> From: Somesh Naidu [mailto:somesh.na...@citrix.com]
>> Sent: 06 January 2015 18:09
>> To: users@cloudstack.apache.org
>> Subject: RE: Xenserver pool HA in Cloudstack
>>
>> Nice article Geoff! I am surprised this information is not present in 
>> standard ACS documentation.
>>
>> -Original Message-
>> From: Geoff Higginbottom [mailto:geoff.higginbot...@shapeblue.com]
>> Sent: Tuesday, January 06, 2015 11:48 AM
>> To: users@cloudstack.apache.org
>> Subject: RE: Xenserver pool HA in Cloudstack
>>
>> @Suresh,
>>
>> The link you referenced is a design doc for a possible future feature, this 
>> is NOT in CloudStack as of today.
>>
>> @Sonali,
>>
>> As promised, here is a link to the article I have been working on today 
>> covering the use of XenServer HA with CloudStack,
>>
>> http://shapeblue.com/cloudstack/xenserver-native-ha-with-cloudstack/
>>
>> In summary if you have installed XS62ESP1004 then you MUST start using 
>> XenServer HA on your XenServer Clusters, if you don’t, and the Pool Master 
>> goes down, no new Pool Master will be elected.
>>
>> Regards
>>
>> Geoff Higginbottom
>>
>> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>>
>> geoff.higginbot...@shapeblue.com
>>
>> -Original Message-
>> From: Suresh Sadhu [mailto:suresh.sa...@citrix.com]
>> Sent: 06 January 2015 16:06
>> To: users@cloudstack.apache.org
>> Subject: RE: Xenserver pool HA in Cloudstack
>>
>> From xen6.2 onwards cloudstack deponds(leverages) on xen native HA 
>> capabilities Please check below link:
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/User+VM+HA+using+native+XS+HA+capabilities
>>
>>
>> regards
>> sadhu
>>
>> -Original Message-
>> From: Somesh Naidu [mailto:somesh.na...@citrix.com]
>> Sent: 06 January 2015 21:25
>> To: users@cloudstack.apache.org
>> Subject: RE: Xenserver pool HA in Cloudstack
>>
>> Sonali, what version of XS are you using?
>>
>> -Original Message-
>> From: Geoff Higginbottom [mailto:geoff.higginbot...@shapeblue.com]
>> Sent: Tuesday, January 06, 2015 9:03 AM
>> To: users@cloudstack.apache.org
>> Subject: RE: Xenserver pool HA in Cloudstack
>>
>> Sonali,
>>
>> There have been some changes recently and as luck would have it today I am 
>> working on a blog article for www.shapeblue.com covering the correct way to 
>> configure HA in XenServer. It's due to go live later today, I'll respond to 
>> this thread once it's live.
>>
>&

RE: Xenserver pool HA in Cloudstack

[Yuri Kogun]

Hi Geoff 

If the pool master is managed outside the cloudstack, does it mean that there 
is no need to set manage.xenserver.pool.master=false when applying patched to 
the Xen cluster with pool HA enabled ?

Will it just  be enough  to unmanage the cluster in cloudstack , disable the 
pool HA on the cluster and proceed with the hotfixes  or should I follow the 
procedure described in the documentation?

Thank you 
Yuri 


> From: geoff.higginbot...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
> Date: Wed, 7 Jan 2015 17:59:53 +
>
> Hi Somesh,
>
> I will be addeing appropriate info from this blog into the official 
> CloudStack Install Docs.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com
>
> -Original Message-
> From: Somesh Naidu [mailto:somesh.na...@citrix.com]
> Sent: 06 January 2015 18:09
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> Nice article Geoff! I am surprised this information is not present in 
> standard ACS documentation.
>
> -Original Message-
> From: Geoff Higginbottom [mailto:geoff.higginbot...@shapeblue.com]
> Sent: Tuesday, January 06, 2015 11:48 AM
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> @Suresh,
>
> The link you referenced is a design doc for a possible future feature, this 
> is NOT in CloudStack as of today.
>
> @Sonali,
>
> As promised, here is a link to the article I have been working on today 
> covering the use of XenServer HA with CloudStack,
>
> http://shapeblue.com/cloudstack/xenserver-native-ha-with-cloudstack/
>
> In summary if you have installed XS62ESP1004 then you MUST start using 
> XenServer HA on your XenServer Clusters, if you don’t, and the Pool Master 
> goes down, no new Pool Master will be elected.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com
>
> -Original Message-
> From: Suresh Sadhu [mailto:suresh.sa...@citrix.com]
> Sent: 06 January 2015 16:06
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> From xen6.2 onwards cloudstack deponds(leverages) on xen native HA 
> capabilities Please check below link:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/User+VM+HA+using+native+XS+HA+capabilities
>
>
> regards
> sadhu
>
> -Original Message-
> From: Somesh Naidu [mailto:somesh.na...@citrix.com]
> Sent: 06 January 2015 21:25
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> Sonali, what version of XS are you using?
>
> -Original Message-
> From: Geoff Higginbottom [mailto:geoff.higginbot...@shapeblue.com]
> Sent: Tuesday, January 06, 2015 9:03 AM
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> Sonali,
>
> There have been some changes recently and as luck would have it today I am 
> working on a blog article for www.shapeblue.com covering the correct way to 
> configure HA in XenServer. It's due to go live later today, I'll respond to 
> this thread once it's live.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com
>
> -Original Message-
> From: Sonali Jadhav [mailto:son...@servercentralen.se]
> Sent: 06 January 2015 13:27
> To: users@cloudstack.apache.org
> Subject: Xenserver pool HA in Cloudstack
>
> Hi,
>
> I have Xenserver cluster setup with 4 Hosts in it. I was wondering about HA 
> configuration of xenserver pool. But at some places I saw that cloudstack HA 
> and Xenserver HA both are different . And One shouldn't use Xenserver HA, 
> since Cloudstack HA is by default active. Is it true?
>
> Yours sincerely,
>
> Sonali Jadhav | System Administrator
> Nordiska Servercentralen
> Direct: (Sweden) +46 7 52472074 | (India) +91 9892837889 Skype ID: 
> sonali.jadhav.sj
> E-mail: son...@servercentralen.se
> Nordiska Servercentralen AB
> Snövitsvägen 5
> S-167 62 Bromma | Sweden
> www.servercentralen.se
> Incredible Servercentral Pvt. Ltd.
> 202, Siddhivinayak Aurum, Near Hyatt Regency Behind Eden Garden, Viman Nagar, 
> 411 014 | Pune, India www.servercentral.in
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build
> CSForge – rapid IaaS deployment framework
> CloudStack Consulting
> CloudStack Software 
> Engineering
> CloudStack Infrastructure 
> Support
> CloudStack Bootcamp Training 
> 

RE: urgent help needed: primary storage became unplugged after xenserver roboot

[Yuri Kogun]

We had a similar problem before. It turned out to be a unsupported combination 
of cloudstack + xenserver (we did run 4.2.1 with xenserver 6.2 sp1) . Please 
check you have all required patches applied on the Xen box. 

Best regards 
Yuri 


> From: cgei...@it1solutions.com
> To: users@cloudstack.apache.org
> Subject: RE: urgent help needed: primary storage became unplugged after 
> xenserver roboot
> Date: Wed, 14 Jan 2015 00:18:54 +
>
> I seem to recal there were some clouds tack files overwritten with updates. 
> I'd try redeploy it in cloudstack. I would also take a pool backup first.
> Also check the XS logs for errors. /var/log/messages and /var/log/SMlog
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
>  Original message 
> From: Yiping Zhang 
> Date: 01/13/2015 5:43 PM (GMT-05:00)
> To: users@cloudstack.apache.org
> Subject: urgent help needed: primary storage became unplugged after xenserver 
> roboot
>
> Hi, All:
>
> I need some urgent help in restoring my CS instance (version 4.3.1).
>
> After patching xenserver 6.2, I rebooted the pool master (I have two xen 
> hypervisors). After the pool master comes back, the SR for the primary 
> storage became “unplugged”, and shown as broken in XenCenter. The volume is 
> not mounted on pool master host any more.
>
> How do I get it back without losing any data ?
>
> Thanks for all helps.
>
> Yiping
>
>
>
  

RE: Root admin system capacity after upgrade to 4.3

[Yuri Kogun]

Tank you, Somesh
It has been running without any problems for a few days now. Will go ahead  
with the production upgrade




> From: somesh.na...@citrix.com
> To: users@cloudstack.apache.org
> Subject: RE: Root admin system capacity after upgrade to 4.3
> Date: Tue, 13 Jan 2015 18:19:10 +
>
> I believe you can ignore this.
>
> The dashboard home page is only supposed to show up some of the elements and 
> when you dig into one of the resources, it shows all the resources for the 
> Zone the resource belongs to.
>
> -----Original Message-
> From: Yuri Kogun [mailto:yko...@outlook.com]
> Sent: Monday, January 12, 2015 4:48 AM
> To: users@cloudstack.apache.org
> Subject: Root admin system capacity after upgrade to 4.3
>
> Hi,
>
> We have upgraded Cloudstack on the test system from 4.2.1 to 4.3.1 , After 
> the upgrade, on the root dashboard in the system capacity I can see only 
> charts for Zone CPU, memory and secondary storage. Before the upgrade we used 
> to have charts on primary storage, Public and shared IP's.
>
> If I click on one of the charts in system capacity , it directs me to Zone 
> details table, where I can see the stats on all resources as expected. Apart 
> from that we don't have any problems with the cloudstack installation and no 
> errors in the logs.
>
> Should I be worried or just ignore this ?
>
> Thank you
> Yuri
>
>
>
>
  

Root admin system capacity after upgrade to 4.3

[Yuri Kogun]

Hi, 

We have upgraded Cloudstack on the test system from 4.2.1 to 4.3.1 , After the 
upgrade,  on the root dashboard in the system capacity I can see only  charts 
for  Zone CPU, memory and secondary storage.  Before the upgrade we used to 
have charts on primary storage, Public and shared IP's. 

If I click on one of the charts in  system capacity , it directs me to  Zone 
details table, where  I can see the stats on all resources as expected. Apart 
from that we don't have any problems with the cloudstack installation and no 
errors in the logs. 

Should I be worried or just ignore this ?

Thank you 
Yuri 



  

RE: Xenserver pool HA in Cloudstack

[Yuri Kogun]

Hi Geoff, 

Great article. Just wanted let you know that  I tested the configuration on my 
test environment  and everything worked as you described, apart from the 
'Recovering from Host Failures' bit. 

In my case the former master  came up after reboot and joined the pool as a 
slave. No manual steps were needed. Below is the extract from the messages on 
the host.


Jan  7 16:53:36 dcv-xenserver-h31 xsconsole: Configure Management Interface 
found no PIFs to present
Jan  7 16:53:42 dcv-xenserver-h31 xsconsole: Configure Management Interface 
found no PIFs to present
Jan  7 16:53:47 dcv-xenserver-h31 xha[6193]: The local host has transitioned to 
online state.
Jan  7 16:53:48 dcv-xenserver-h31 xha[6193]: propose_master returns FALSE.
Jan  7 16:53:48 dcv-xenserver-h31 xha[6193]: propose_master returns FALSE.
Jan  7 16:53:48 dcv-xenserver-h31 xapi: [ info|dcv-xenserver-h31|0 
thread_zero|Checking HA configuration D:e685fe6c5de9|xapi_ha] ha_propose_master 
failed: looking for another master
Jan  7 16:53:48 dcv-xenserver-h31 xapi: [ info|dcv-xenserver-h31|0 
thread_zero|Checking HA configuration D:e685fe6c5de9|xapi_ha] This node will 
become the slave of host 17aa5709-ec88-4e9a-93ad-0adc1b43c32f (192.168.56.30)
Jan  7 16:53:49 dcv-xenserver-h31 xsconsole: Local shell
Jan  7 16:53:58 dcv-xenserver-h31 xapi: [ warn|dcv-xenserver-h31|17||xapi_fuse] 
Caught an exception flushing database (perhaps it hasn't been initialised yet): 
INTERNAL_ERROR: [ Failure("hd") ]; restarting immediately


It could be because I have applied patches up to  XS62E014 and  something was 
fixed. 

Best regards 
Yuri 




> From: geoff.higginbot...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
> Date: Tue, 6 Jan 2015 16:48:19 +
>
> @Suresh,
>
> The link you referenced is a design doc for a possible future feature, this 
> is NOT in CloudStack as of today.
>
> @Sonali,
>
> As promised, here is a link to the article I have been working on today 
> covering the use of XenServer HA with CloudStack,
>
> http://shapeblue.com/cloudstack/xenserver-native-ha-with-cloudstack/
>
> In summary if you have installed XS62ESP1004 then you MUST start using 
> XenServer HA on your XenServer Clusters, if you don’t, and the Pool Master 
> goes down, no new Pool Master will be elected.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com
>
> -Original Message-
> From: Suresh Sadhu [mailto:suresh.sa...@citrix.com]
> Sent: 06 January 2015 16:06
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> From xen6.2 onwards cloudstack deponds(leverages) on xen native HA 
> capabilities Please check below link:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/User+VM+HA+using+native+XS+HA+capabilities
>
>
> regards
> sadhu
>
> -Original Message-
> From: Somesh Naidu [mailto:somesh.na...@citrix.com]
> Sent: 06 January 2015 21:25
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> Sonali, what version of XS are you using?
>
> -Original Message-
> From: Geoff Higginbottom [mailto:geoff.higginbot...@shapeblue.com]
> Sent: Tuesday, January 06, 2015 9:03 AM
> To: users@cloudstack.apache.org
> Subject: RE: Xenserver pool HA in Cloudstack
>
> Sonali,
>
> There have been some changes recently and as luck would have it today I am 
> working on a blog article for www.shapeblue.com covering the correct way to 
> configure HA in XenServer. It's due to go live later today, I'll respond to 
> this thread once it's live.
>
> Regards
>
> Geoff Higginbottom
>
> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>
> geoff.higginbot...@shapeblue.com
>
> -Original Message-
> From: Sonali Jadhav [mailto:son...@servercentralen.se]
> Sent: 06 January 2015 13:27
> To: users@cloudstack.apache.org
> Subject: Xenserver pool HA in Cloudstack
>
> Hi,
>
> I have Xenserver cluster setup with 4 Hosts in it. I was wondering about HA 
> configuration of xenserver pool. But at some places I saw that cloudstack HA 
> and Xenserver HA both are different . And One shouldn't use Xenserver HA, 
> since Cloudstack HA is by default active. Is it true?
>
> Yours sincerely,
>
> Sonali Jadhav | System Administrator
> Nordiska Servercentralen
> Direct: (Sweden) +46 7 52472074 | (India) +91 9892837889 Skype ID: 
> sonali.jadhav.sj
> E-mail: son...@servercentralen.se
> Nordiska Servercentralen AB
> Snövitsvägen 5
> S-167 62 Bromma | Sweden
> www.servercentralen.se
> Incredible Servercentral Pvt. Ltd.
> 202, Siddhivinayak Aurum, Near Hyatt Regency Behind Eden Garden, Viman Nagar, 
> 411 014 | Pune, India www.servercentral.in
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build

RE: Cloudmonkey 5.3.0 - problems connecting with self signed certificate.

[Yuri Kogun]

Hi Rohit 
I have tried to get the page directly from python console  with and without 
verify and it worked with verify=False. So there is no problems with openssl or 
requests library on the VM 

Please see the output below. 


Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import requests
>>> r = requests.get('https://localhost:6443/client')
.
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL 
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

>>> r = requests.get('https://localhost:6443/client',verify=False)
/usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py:734:
 InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
certificate verification is strongly advised. See: 
https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)


I checked the source code on cloudmonkey and it looks like the problem is in 
the login routine line 72  in the requests.py module in Cloudmonkey. The 
Session.post call does not have verify parameter passed to it. I guess it will 
only affect users trying to login with username/password authentication. 


def login(url, username, password, domain="/"):
    """
    Login and obtain a session to be used for subsequent API calls
    Wrong username/password leads to HTTP error code 531
    """
    args = {}

    args["command"] = 'login'
    args["username"] = username
    args["password"] = password
    args["domain"] = domain
    args["response"] = "json"

    sessionkey = ''
    session = requests.Session()

    try:
        resp = session.post(url, params=args)



Best regards 
Yuri 




> Date: Wed, 3 Dec 2014 20:52:42 +0530
> From: rohit.ya...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: Re: Cloudmonkey 5.3.0 - problems connecting with self signed 
> certificate.
>
> Hi Yuri,
>
> Sorry I think I missed that earlier, cloudmonkey uses requests library
> to make HTTP/S requests; we pass the verifysslcert option to it. If
> you've set it to false, then when making requests using "requests"
> library/module we pass it false. You may check the python code and debug
> why that is happening. It's a dirty fix, ideally you should make sure
> you've valid SSL certificate.
>
> Check this out for reference:
> http://stackoverflow.com/questions/10667960/python-requests-throwing-up-sslerror
>
> Lastly, can you try to upgrade requests and openssl? And share any
> further details which may help fix this issue. Thanks.
>
> On Wednesday 03 December 2014 07:17 PM, Yuri Kogun wrote:
>> Thanks for reply, Rohit
>> This is exactly what I am doing, maybe it is not clear from the previous 
>> email. Below is the output from the log
>>
>> (local) 🐵> set verifysslcert false(local) 🐵> list zonesConnection refused by 
>> server: [Errno 1] _ssl.c:492: error:14090086:SSL 
>> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
>> Authentication failed(local) 🐵>
>>
>> Below is the output from the log file:
>> 2014-12-03 13:44:53,023 - cloudmonkey.py:398 - [DEBUG] Invalid command 
>> result: None2014-12-03 13:46:05,404 - cloudmonkey.py:98 - [DEBUG] Loaded 
>> config fields:['profile=local', 'asyncblock=true', 'paramcompletion=true', 
>> 'history_file=/root/.cloudmonkey/history', 
>> 'cache_file=/root/.cloudmonkey/cache', 'log_file=/root/.cloudmonkey/log', 
>> 'color=true', 'prompt=(local) \xf0\x9f\x90\xb5> ', 'display=default', 
>> 'username=admin', 'domain=/', 'apikey=', 
>> 'url=https://192.168.56.11:6443/client/api', 'expires=600', 'secretkey=', 
>> 'timeout=3600', 'password=password', 'verifysslcert=false']2014-12-03 
>> 13:46:09,051 - requester.py:46 - [DEBUG]  START Request 
>> 2014-12-03 13:46:09,053 - requester.py:46 - [DEBUG] Requesting 
>> command=listZones, args={}2014-12-03 13:46:09,079 - connectionpool.py:700 - 
>> [INFO] Starting new HTTPS connection (1): 192.168.56.112014-12-03 
>> 13:46:09,103 - requester.py:46 - [DEBUG]  END Request 
>> 2014-12-03 13:46:09,108 - cloudmonkey.py:398 - [DEBUG] Invalid command 
>> result: None
>>
>> Best regards Yuri
>>> From: rohit

RE: Cloudmonkey 5.3.0 - problems connecting with self signed certificate.

[Yuri Kogun]

Thank you, Rohit.  The latest commit works for me. 

Best regards 
Yuri 


> Date: Wed, 3 Dec 2014 23:12:33 +0530
> From: rohit.ya...@shapeblue.com
> To: yko...@outlook.com; users@cloudstack.apache.org
> Subject: Re: Cloudmonkey 5.3.0 - problems connecting with self signed 
> certificate.
>
> Hi Yuri,
>
> Thanks for sharing the analysis, I've fixed the issue on master branch
> (91afdd318f4ecba13963742f4c0bf28748093ff9). This was a corner case that
> I forgot to consider.
>
> Thanks for your contribution, I'll kickstart CloudMonkey 5.3.1 bugfix
> release soon but meanwhile you may download and install cloudmonkey from
> the git repo's master branch.
>
> On Wednesday 03 December 2014 10:09 PM, Yuri Kogun wrote:
>> Hi Rohit
>> I have tried to get the page directly from python console with and without 
>> verify and it worked with verify=False. So there is no problems with openssl 
>> or requests library on the VM
>>
>> Please see the output below.
>>
>>
>> Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)
>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>
>>>>> import requests
>>>>> r = requests.get('https://localhost:6443/client')
>> .
>> raise SSLError(e, request=request)
>> requests.exceptions.SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL 
>> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
>>
>>>>> r = requests.get('https://localhost:6443/client',verify=False)
>> /usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py:734:
>>  InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
>> certificate verification is strongly advised. See: 
>> https://urllib3.readthedocs.org/en/latest/security.html
>> InsecureRequestWarning)
>>
>>
>> I checked the source code on cloudmonkey and it looks like the problem is in 
>> the login routine line 72 in the requests.py module in Cloudmonkey. The 
>> Session.post call does not have verify parameter passed to it. I guess it 
>> will only affect users trying to login with username/password authentication.
>>
>>
>> def login(url, username, password, domain="/"):
>> """
>> Login and obtain a session to be used for subsequent API calls
>> Wrong username/password leads to HTTP error code 531
>> """
>> args = {}
>>
>> args["command"] = 'login'
>> args["username"] = username
>> args["password"] = password
>> args["domain"] = domain
>> args["response"] = "json"
>>
>> sessionkey = ''
>> session = requests.Session()
>>
>> try:
>> resp = session.post(url, params=args)
>>
>>
>>
>> Best regards
>> Yuri
>>
>>
>>
>> 
>>> Date: Wed, 3 Dec 2014 20:52:42 +0530
>>> From: rohit.ya...@shapeblue.com
>>> To: users@cloudstack.apache.org
>>> Subject: Re: Cloudmonkey 5.3.0 - problems connecting with self signed 
>>> certificate.
>>>
>>> Hi Yuri,
>>>
>>> Sorry I think I missed that earlier, cloudmonkey uses requests library
>>> to make HTTP/S requests; we pass the verifysslcert option to it. If
>>> you've set it to false, then when making requests using "requests"
>>> library/module we pass it false. You may check the python code and debug
>>> why that is happening. It's a dirty fix, ideally you should make sure
>>> you've valid SSL certificate.
>>>
>>> Check this out for reference:
>>> http://stackoverflow.com/questions/10667960/python-requests-throwing-up-sslerror
>>>
>>> Lastly, can you try to upgrade requests and openssl? And share any
>>> further details which may help fix this issue. Thanks.
>>>
>>> On Wednesday 03 December 2014 07:17 PM, Yuri Kogun wrote:
>>>> Thanks for reply, Rohit
>>>> This is exactly what I am doing, maybe it is not clear from the previous 
>>>> email. Below is the output from the log
>>>>
>>>> (local) 🐵> set verifysslcert false(local) 🐵> list zonesConnection refused 
>>>> by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
>>>> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
>>>>

RE: Cloudmonkey 5.3.0 - problems connecting with self signed certificate.

[Yuri Kogun]

Thanks for reply,  Rohit 
This is exactly what I am doing, maybe it is not clear from the previous email. 
Below is the output from the log 

(local) 🐵 > set verifysslcert false(local) 🐵 > list zonesConnection refused by 
server: [Errno 1] _ssl.c:492: error:14090086:SSL 
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
Authentication failed(local) 🐵 >

Below is the output from the log file:
2014-12-03 13:44:53,023 - cloudmonkey.py:398 - [DEBUG] Invalid command result: 
None2014-12-03 13:46:05,404 - cloudmonkey.py:98 - [DEBUG] Loaded config 
fields:['profile=local', 'asyncblock=true', 'paramcompletion=true', 
'history_file=/root/.cloudmonkey/history', 
'cache_file=/root/.cloudmonkey/cache', 'log_file=/root/.cloudmonkey/log', 
'color=true', 'prompt=(local) \xf0\x9f\x90\xb5 > ', 'display=default', 
'username=admin', 'domain=/', 'apikey=', 
'url=https://192.168.56.11:6443/client/api', 'expires=600', 'secretkey=', 
'timeout=3600', 'password=password', 'verifysslcert=false']2014-12-03 
13:46:09,051 - requester.py:46 - [DEBUG]  START Request 
2014-12-03 13:46:09,053 - requester.py:46 - [DEBUG] Requesting 
command=listZones, args={}2014-12-03 13:46:09,079 - connectionpool.py:700 - 
[INFO] Starting new HTTPS connection (1): 192.168.56.112014-12-03 13:46:09,103 
- requester.py:46 - [DEBUG]  END Request 
2014-12-03 13:46:09,108 - cloudmonkey.py:398 - [DEBUG] Invalid command result: 
None

Best regards Yuri
> From: rohit.ya...@shapeblue.com
> To: yko...@outlook.com
> CC: users@cloudstack.apache.org
> Subject: Re: Cloudmonkey 5.3.0 - problems connecting with  self signed 
> certificate.
> Date: Wed, 3 Dec 2014 13:37:11 +
> 
> Hi Yuri,
> 
> For the specific server profile, there is some issue with SSL cert. You can 
> turn off ssl cert checking in cloudmonkey using:
> 
> set verifysslcert false
> 
> This setting is per server profile.
> 
> This is also documented on the wiki here, for your reference:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI
> 
> Hope this helps.
> 
> > On 03-Dec-2014, at 7:00 pm, Yuri Kogun  wrote:
> >
> > I know this supposed to be fixed in 5.3.0 but it does not work for me. Can 
> > somebody point what I am doing wrong.
> > I have fresh cloudstack 4.2.1-6 setup configured to listen on https port 
> > 6443.  When trying to connect to the server using cloudmonkey I am getting 
> > the following error
> >
> > ☁ Apache CloudStack 🐵 cloudmonkey 5.3.0. Type help or ? to list commands.
> >
> > Using management server profile: local
> >
> > (local) 🐵 > set url https://192.168.56.11:6443/client/api
> > (local) 🐵 > list zones
> > Connection refused by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
> > routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
> > Error Authentication failed
> > (local) 🐵 > set verifysslcert False
> > (local) 🐵 > list zones
> > Connection refused by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
> > routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
> > Error Authentication failed
> > (local) 🐵 >  cloudmonkey
> >
> >
> > Best regards
> > Yuri
> 
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +91 88 262 30892 | rohit.ya...@shapeblue.com
> Blog: bhaisaab.org | Twitter: @_bhaisaab
> 
> 
> 
> Find out more about ShapeBlue and our range of CloudStack related services
> 
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software 
> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure 
> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training 
> Courses<http://shapeblue.com/cloudstack-training/>
> 
> This email and any attachments to it may be confidential and are intended 
> solely for the use of the individual to whom it is addressed. Any views or 
> opinions expressed are solely those of the author and do not necessarily 
> represent those of Shape Blue Ltd or related companies. If you are not the 
> intended recipient of this email, you must neither take any action based upon 
> its contents, nor copy or show it to anyone. Please contact the sender if you 
> believe you have received this email in error. Shape Blue Ltd is a company 
> incorporated in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue Ltd. 
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
> operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company 
> registered by The Republic of South Africa and is traded under license from 
> Shape Blue Ltd. ShapeBlue is a registered trademark.
  

Cloudmonkey 5.3.0 - problems connecting with self signed certificate.

[Yuri Kogun]

I know this supposed to be fixed in 5.3.0 but it does not work for me. Can 
somebody point what I am doing wrong. I have fresh cloudstack 4.2.1-6 setup 
configured to listen on https port 6443.  When trying to connect to the server 
using cloudmonkey I am getting the following error 
☁ Apache CloudStack 🐵 cloudmonkey 5.3.0. Type help or ? to list commands.
Using management server profile: local
(local) 🐵 > set url https://192.168.56.11:6443/client/api(local) 🐵 > list 
zonesConnection refused by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
Authentication failed(local) 🐵 > set verifysslcert False(local) 🐵 > list 
zonesConnection refused by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
Authentication failed(local) 🐵 >  cloudmonkey

Best regards Yuri 

Cloudmonkey 5.3.0 - problems connecting with self signed certificate.

[Yuri Kogun]

I know this supposed to be fixed in 5.3.0 but it does not work for me. Can 
somebody point what I am doing wrong. I have fresh cloudstack 4.2.1-6 setup 
configured to listen on https port 6443.  When trying to connect to the server 
using cloudmonkey I am getting the following error 
☁ Apache CloudStack 🐵 cloudmonkey 5.3.0. Type help or ? to list commands.
Using management server profile: local
(local) 🐵 > set url https://192.168.56.11:6443/client/api(local) 🐵 > list 
zonesConnection refused by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
Authentication failed(local) 🐵 > set verifysslcert False(local) 🐵 > list 
zonesConnection refused by server: [Errno 1] _ssl.c:492: error:14090086:SSL 
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedError 
Authentication failed(local) 🐵 >  cloudmonkey

Best regards Yuri 

RE: CloudStack Guest Scripts and "cloud-init"

[Yuri Kogun]

Hi, 
What version of cloudinit packages are you using? The cloudstack support has 
been added in the version 0.7.1

Please check this link  https://bugzilla.redhat.com/show_bug.cgi?id=907547

You might need to update your RPM. The EPEL repository still have 0.6.3  
version 

Best regards
Yuri 


> From: shanker.ba...@shapeblue.com
> To: users@cloudstack.apache.org
> Subject: CloudStack Guest Scripts and "cloud-init"
> Date: Mon, 14 Oct 2013 13:18:42 +
> 
> Hi Guys,
> 
> Anyone using cloud-init with CloudStack VMs successfully? I tried to use 
> cloud-init
> from the EPEL repos on CentOS and it seemed to assume an AWS environment.
> 
> Thought I would check before I take a 2nd look. I am looking at the 
> feasibility of
> using cloud-init for the providing the cloudstack-guest-set-* script 
> functionality.
> 
> --
> @shankerbalan
> 
> M: +91 98860 60539 | O: +91 (80) 67935867
> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
> Bangalore - 560 055
> 
> CloudStack Bootcamp Training on 27/28 November, Bangalore
> http://www.shapeblue.com/cloudstack-training/
> 
> 
> 
> 
> This email and any attachments to it may be confidential and are intended 
> solely for the use of the individual to whom it is addressed. Any views or 
> opinions expressed are solely those of the author and do not necessarily 
> represent those of Shape Blue Ltd or related companies. If you are not the 
> intended recipient of this email, you must neither take any action based upon 
> its contents, nor copy or show it to anyone. Please contact the sender if you 
> believe you have received this email in error. Shape Blue Ltd is a company 
> incorporated in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue Ltd. 
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
> operated under license from Shape Blue Ltd. ShapeBlue is a registered 
> trademark.