Can not create the ceph RBD primary storage?

[杜焱霖]

Hi，Guys:


  I am using ceph as my cloudstack cluster storage, But I can not add ceph RBD 
pool to my cloudstack cluster.  
  The related info is as below:


The label.rados.monitor is 10.29.44.1:6789 , The kvm node and the v-xxx-VM and 
s-xxx-VM can access the ceph node.
kvm001:~# telnet 10.29.44.1 6789
Trying 10.29.44.1...
Connected to 10.29.44.1.
Escape character is '^]'.
ceph v027 
, 
ell


root@s-252-VM:~#  telnet 10.29.44.1 6789
Trying 10.29.44.1...
Connected to 10.29.44.1.
Escape character is '^]'.
ceph v027 
, 
 


The RBD pool is exists: cloudstack

The user cloudstack has the access to the RBD pool: cloudstack

The secret is 'AQDLxqlhIdOLJRAABPqps8O6eSGbFnyR7aSJwQ==' that has no / (slash) 
in the secret.

The KVM node has ceph config: ceph.client.admin.keyring.

# ls /etc/ceph/
ceph.client.admin.keyring  ceph.client.cloudstack.keyring  ceph.conf  rbdmap


The related info and The libvirt secrets has nothing:
# virsh pool-list
 Name   StateAutostart

 60b59087-7c53-3058-a50c-f50737e556bc   active   no  
 c4355ed4-8833-381f-b3f7-2981782ee3fa   active   no
 c8e9ca6a-c004-3851-a074-19f4948b28ff   active   no
 d8dabcb0-1a57-4e13-8a82-339b2052dec1   active   no

# virsh secret-list
 UUID   Usage
---

# ls -a /etc/libvirt/secrets/
.  ..



But I can not find the storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 on 
cloudstack UI and the storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 will 
change when I reclick the add primary stotage button.

After checked all the config, I restart the management-server and 
cloudstack-agent services. The error is still the same: 
org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 
'cloudstack' exist?: No such file or directory 

2021-12-06 17:54:17,921 DEBUG [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:96eddfd2) 
b90eae9d-973c-362c-8afc-af88f0743892
b90eae9d-973c-362c-8afc-af88f0743892


cloudstack






2021-12-06 17:54:39,461 DEBUG [kvm.resource.LibvirtComputingResource] 
(UgentTask-5:null) (logid:) Executing: 
/usr/share/cloudstack-common/scripts/vm/network/security_group.py 
get_rule_logs_for_vms 
2021-12-06 17:54:39,463 DEBUG [kvm.resource.LibvirtComputingResource] 
(UgentTask-5:null) (logid:) Executing while with timeout : 180
2021-12-06 17:54:39,534 DEBUG [kvm.resource.LibvirtComputingResource] 
(UgentTask-5:null) (logid:) Execution is successful.
2021-12-06 17:54:39,535 DEBUG [kvm.resource.LibvirtConnection] 
(UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system
2021-12-06 17:54:39,551 DEBUG [cloud.agent.Agent] (UgentTask-5:null) (logid:) 
Sending ping: Seq 15-6:  { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"v-255-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"v-249-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"s-250-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"},"r-254-VM":{"state":"PowerOn","host":"whdckvm002.cn.prod"}},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"15","wait":"0","bypassHostMaintenance":"false"}}]
 }
2021-12-06 17:54:39,620 DEBUG [cloud.agent.Agent] (Agent-Handler-2:null) 
(logid:) Received response: Seq 15-6:  { Ans: , MgmtId: 345052215515, via: 15, 
Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":"15","wait":"0","bypassHostMaintenance":"false"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
 }
2021-12-06 17:54:47,958 ERROR [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:96eddfd2) Failed to create RBD storage 
pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the 
pool 'cloudstack' exist?: No such file or directory
2021-12-06 17:54:47,959 ERROR [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:96eddfd2) Failed to create the RBD storage 
pool, cleaning up the libvirt secret
2021-12-06 17:54:47,961 WARN  [cloud.agent.Agent] (agentRequest-Handler-4:null) 
(logid:96eddfd2) Caught: 
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: 
b90eae9d-973c-362c-8afc-af88f0743892
at 
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645)
at 
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329)
at 
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323)
at 
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
at 
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
at 
com.cloud.hy

[GitHub] [cloudstack-terraform-provider] harikrishna-patnala closed issue #16: How to get the source nat ip address from a network?

[GitBox]

harikrishna-patnala closed issue #16:
URL: https://github.com/apache/cloudstack-terraform-provider/issues/16


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] harikrishna-patnala commented on issue #16: How to get the source nat ip address from a network?

[GitBox]

harikrishna-patnala commented on issue #16:
URL: 
https://github.com/apache/cloudstack-terraform-provider/issues/16#issuecomment-987622642


   Great, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] hrak commented on issue #16: How to get the source nat ip address from a network?

[GitBox]

hrak commented on issue #16:
URL: 
https://github.com/apache/cloudstack-terraform-provider/issues/16#issuecomment-987617391


   Yes, it can be closed, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] harikrishna-patnala commented on issue #26: cloudstack_port_forward doesn't seem to be working

[GitBox]

harikrishna-patnala commented on issue #26:
URL: 
https://github.com/apache/cloudstack-terraform-provider/issues/26#issuecomment-987608709


   There is another issue with error #21 , closing this one


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] harikrishna-patnala closed issue #26: cloudstack_port_forward doesn't seem to be working

[GitBox]

harikrishna-patnala closed issue #26:
URL: https://github.com/apache/cloudstack-terraform-provider/issues/26


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] harikrishna-patnala commented on issue #16: How to get the source nat ip address from a network?

[GitBox]

harikrishna-patnala commented on issue #16:
URL: 
https://github.com/apache/cloudstack-terraform-provider/issues/16#issuecomment-987601245


   Hi @hrak, can we close this issue if there are no further questons


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] vladimirpetrov opened a new issue #26: cloudstack_port_forward doesn't seem to be working

[GitBox]

vladimirpetrov opened a new issue #26:
URL: https://github.com/apache/cloudstack-terraform-provider/issues/26


   I have the following resource declaration:
   
   ```
   resource "cloudstack_port_forward" "pf1" {
 ip_address_id = "9fae134f-7fba-4c1d-beaa-698ecd13c71b"
   
 forward {
   protocol   = "tcp"
   private_port   = 22
   public_port= 22
   virtual_machine_id = cloudstack_instance.VM1.id
 }
   }
   ```
   
   I have the VM already deployed and isolated network implemented, but still 
there is a strange error message when trying to apply this configuration:
   ```
   │ Error: insufficient items for attribute "forward"; must have at least 1
   │ 
   │   with cloudstack_port_forward.pf1,
   │   on cloudstack_port_forward.tf line 1, in resource 
"cloudstack_port_forward" "pf1":
   │1: resource "cloudstack_port_forward" "pf1" {
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-terraform-provider] vladimirpetrov opened a new issue #25: cloudstack_template does not wait until the template is ready for use

[GitBox]

vladimirpetrov opened a new issue #25:
URL: https://github.com/apache/cloudstack-terraform-provider/issues/25


   When you have both upload template and deploy VM operations, the 
cloudstack_template does not really wait until the template is in 'Ready' state 
so the next operation - deploy VM - fails:
   
   ```
   cloudstack_template.tiny_linux: Creating...
   cloudstack_instance.VM1: Creating...
   cloudstack_template.tiny_linux: Still creating... [10s elapsed]
   cloudstack_template.tiny_linux: Creation complete after 11s 
[id=e375a36e-64e8-48a3-b563-68aa70c56295]
   ╷
   │ Error: Error retrieving ID of template Tiny Linux: No match found for Tiny 
Linux: &{Count:0 Templates:[]}
   │ 
   │   with cloudstack_autoscale_vm_profile.vm_scale,
   │   on cloudstack_autoscale_vm_profile.tf line 1, in resource 
"cloudstack_autoscale_vm_profile" "vm_scale":
   │1: resource "cloudstack_autoscale_vm_profile" "vm_scale" {
   │ 
   ╵
   ╷
   │ Error: Error creating the new instance TERRAFORMVM1: CloudStack API error 
530 (CSExceptionErrorCode: 4250): Template 205 has not been completely 
downloaded to zone 1
   │ 
   │   with cloudstack_instance.VM1,
   │   on cloudstack_instance.tf line 1, in resource "cloudstack_instance" 
"VM1":
   │1: resource "cloudstack_instance" "VM1" {
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Live migration between AMD Epyc and Ubuntu 18.04 and 20.04

[Gabriel Bräscher]

Hi Paul (& all),

I strongly believe that this is a bug in QEMU.
I was looking for bugs and found something that looks related to what we
are seeing. Precisely at Ubuntu's bug #*1887490*
:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1887490

In the link above, there was the following comment:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1887490/comments/53

It seems one of the patches also introduced a regression:*
lp-1887490-cpu_map-Add-missing-AMD-SVM-features.patchadds various
SVM-related flags. Specifically npt and nrip-save are now expected to be
present by default as shown in the updated testdata.This however breaks
migration from instances using *EPYC* or *EPYC-IBPB* CPU models started
with libvirt versions prior to this one because the instance on the target
host has these extra flags


More about #*1887490*
 can be found
at the mail
https://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg5842376.html.
We can see that the specific bug was addressed in "linux (5.4.0-49.53)
focal".

linux (5.4.0-49.53) focal; urgency=medium

  * Add/Backport EPYC-v3 and EPYC-Rome CPU model (LP: #1887490)
- kvm: svm: Update svm_xsaves_supported


Regards,
Gabriel.

On Fri, Dec 3, 2021 at 10:59 AM Paul Angus 
wrote:

> Which version(s) of QEMU are you using Wido?
>
> We've just be upgrading CentOS 7.6 to 7.9
> Most 7.6 hosts had qemu-ev 2.10 on it  (the buggy one). 2.12 was on the
> new hosts.
> We were getting errors complaining that the ibpb CPU feature wasn't
> available when migrating to the updated OS hosts (even though identical
> hardware).
>
> Upgrading qemu-ev to 2.12 on the originating host, then stopping and
> starting the VMs, then allowed us to migrate.  We couldn't find any
> solution that didn't involve stopping and starting the VMs.
>
> Paul.
>
> -Original Message-
> From: Wido den Hollander 
> Sent: Monday, November 29, 2021 7:57 AM
> To: dev@cloudstack.apache.org; Wei ZHOU 
> Subject: Re: Live migration between AMD Epyc and Ubuntu 18.04 and 20.04
>
>
>
> On 11/24/21 10:36 PM, Wei ZHOU wrote:
> > Hi Wido,
> >
> > I think it is not good to run an environment with two ubuntu/qemu
> versions.
> > It always happens that some cpu features are supported in the higher
> > version but not supported in the older version.
> > From my experience, the migration from older version to higher version
> > works like a charm, but there were many issues in migration from
> > higher version to older version.
> >
>
> I understand. But with a large amount of hosts and working your way
> through upgrades you sometimes run into these situations. Therefor it would
> be welcome if it works.
>
> > I do not have a solution for you. I have tried to hack
> > /etc/libvirt/hooks/qemu but it didn't work.
> > Have you tried with other cpu models like x86_Opteron_G5 ? you can
> > find the cpu features of each cpu model in /usr/share/libvirt/cpu_map/
> >
>
> I have not tried that yet, but I can see if that works.
>
> The EPYC-IBPB CPU model is identical on 18.04 and 20.04, but even using
> that model we can't seem to migrate as it complains about the 'npt' feature.
>
> Wido
>
> > Anyway, even if the vm migration succeeds, you do not know if vm works
> > fine. I believe the best solution is upgrading all hosts to the same
> > OS version.
> >
> > -Wei
> >
> > On Tue, 23 Nov 2021 at 16:31, Wido den Hollander  wrote:
> >
> >> Hi,
> >>
> >> I'm trying to debug an issue with live migrations between Ubuntu
> >> 18.04 and 20.04 machines each with different CPUs:
> >>
> >> - Ubuntu 18.04 with AMD Epyc 7552 (Rome)
> >> - Ubuntu 20.04 with AMD Epyc 7662 (Milan)
> >>
> >> We are currently using this setting:
> >>
> >> guest.cpu.mode=custom
> >> guest.cpu.model=EPYC
> >>
> >> This does not allow for live migrations:
> >>
> >> Ubuntu 20.04 with Epyc 7662 to Ubuntu 18.04 with Epyc 7552 fails
> >>
> >> "ExecutionException : org.libvirt.LibvirtException: unsupported
> >> configuration: unknown CPU feature: npt"
> >>
> >> So we tried to define a set of features manually:
> >>
> >> guest.cpu.features=3dnowprefetch abm adx aes apic arat avx avx2 bmi1
> >> bmi2 clflush clflushopt cmov cr8legacy cx16 cx8 de f16c fma fpu
> >> fsgsbase fxsr fxsr_opt lahf_lm lm mca mce misalignsse mmx mmxext
> >> monitor movbe msr mtrr nx osvw pae pat pclmuldq pdpe1gb pge pni
> >> popcnt pse pse36 rdrand rdseed rdtscp sep sha-ni smap smep sse sse2
> >> sse4.1 sse4.2 sse4a
> >> ssse3 svm syscall tsc vme xgetbv1 xsave xsavec xsaveopt -npt -x2apic
> >> -hypervisor -topoext -nrip-save
> >>
> >> This results in this going into the XML:
> >>
> >> 
> >>
> >> You would say that works, but then the target host (18.04 with the
> >> 7552) says it doesn't support the feature 'npt' and the migration still
> fails.
> >>
> >> Now we could ofcourse use the kvm64 CPU from Qemu, but that's lacking
> >> so many features that for example TLS offl

[GitHub] [cloudstack-terraform-provider] vladimirpetrov opened a new issue #24: Instance cannot be expunged, only destroyed

[GitBox]

vladimirpetrov opened a new issue #24:
URL: https://github.com/apache/cloudstack-terraform-provider/issues/24


   When invoking the 'destroy' command to a running VM, it cannot be expunged 
as well. Which causes other issues, for example the VM's network cannot be 
destroyed too since it's still connected to the destroyed VM.
   
   ```
   Error: Error deleting network TERRAFORM-isolated1: Undefined error: 
{"errorcode":530,"errortext":"Failed to delete network"}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org