Re: Issues with libvirt after upgrading to RockyLinux 8.5

[Jeremy Hansen]

But if I convert all the hosts to Rocky and upgrade to 4.16, I should be ok?

Thanks

> On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva 
> mailto:slav...@storpool.com.INVALID)> wrote:
> Sorry, I didn't pay attention to your CS version. After the upgrade, I
> think you will have the same problem. Because in the DB, there is
> information about host/hosts on this cluster that is/are with CentOS.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen 
> wrote:
>
> > I noticed in the compatibility matrix that Rocky isn’t supported until
> > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still
> > going to complain about the centos/rocky mix? If I convert all my existing
> > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t
> > CentOS and Rocky be considered that same thing… sort of…?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> >
> > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> > slav...@storpool.com.INVALID> wrote:
> > Hi Jeremy,
> >
> > It will help if you have another cluster for Rocky Linux. Hosts need to be
> > of the same OS, it's not possible to mix OSes in the same cluster.
> >
> > Best regards,
> > Slavka
> >
> > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen 
> > wrote:
> >
> > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> > and qemu and ca-certificates to the same version as the other functional
> > nodes. That didn’t seem to help. This is obviously an ssl issue but I
> > don’t really know what to do about it.
> >
> > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 

Re: Issues with libvirt after upgrading to RockyLinux 8.5

[Slavka Peleva]

Sorry, I didn't pay attention to your CS version. After the upgrade, I
think you will have the same problem. Because in the DB, there is
information about host/hosts on this cluster that is/are with CentOS.

Best regards,
Slavka

On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen 
wrote:

> I noticed in the compatibility matrix that Rocky isn’t supported until
> 4.16.0.0.  If I upgrade Cloudstack first, would this help or is it still
> going to complain about the centos/rocky mix?  If I convert all my existing
> nodes to Rocky, which is the plan anyway, will this go away?  Shouldn’t
> CentOS and Rocky be considered that same thing… sort of…?
>
> Thanks
> -jeremy
>
>
>
>
> On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> slav...@storpool.com.INVALID> wrote:
> Hi Jeremy,
>
> It will help if you have another cluster for Rocky Linux. Hosts need to be
> of the same OS, it's not possible to mix OSes in the same cluster.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen 
> wrote:
>
> Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> and qemu and ca-certificates to the same version as the other functional
> nodes. That didn’t seem to help. This is obviously an ssl issue but I
> don’t really know what to do about it.
>
> 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
>
> I also noticed this:
>
> 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-19617:ctx-e0dfdb18) 

Re: Issues with libvirt after upgrading to RockyLinux 8.5

[Jeremy Hansen]

I noticed in the compatibility matrix that Rocky isn’t supported until 
4.16.0.0. If I upgrade Cloudstack first, would this help or is it still going 
to complain about the centos/rocky mix? If I convert all my existing nodes to 
Rocky, which is the plan anyway, will this go away? Shouldn’t CentOS and Rocky 
be considered that same thing… sort of…?

Thanks
-jeremy

> On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva 
> mailto:slav...@storpool.com.INVALID)> wrote:
> Hi Jeremy,
>
> It will help if you have another cluster for Rocky Linux. Hosts need to be
> of the same OS, it's not possible to mix OSes in the same cluster.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen 
> wrote:
>
> > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> > and qemu and ca-certificates to the same version as the other functional
> > nodes. That didn’t seem to help. This is obviously an ssl issue but I
> > don’t really know what to do about it.
> >
> > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> >
> > I also noticed this:
> >
> > 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle
> > host connection:
> > java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> > hostOS: Rocky into a 

Re: Issues with libvirt after upgrading to RockyLinux 8.5

[Slavka Peleva]

Hi Jeremy,

It will help if you have another cluster for Rocky Linux. Hosts need to be
of the same OS, it's not possible to mix OSes in the same cluster.

Best regards,
Slavka

On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen 
wrote:

> Any tips on how I would troubleshoot this?  I’ve tried downgrading libvirt
> and qemu and ca-certificates to the same version as the other functional
> nodes.  That didn’t seem to help.  This is obviously an ssl issue but I
> don’t really know what to do about it.
>
> 2021-12-15 18:04:14,438 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-15 18:04:14,444 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-15 18:04:14,447 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:14,466 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-15 18:04:14,467 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-15 18:04:14,471 INFO  [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-15 18:04:14,485 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-15 18:04:14,592 INFO  [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-15 18:04:14,663 INFO  [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-15 18:04:14,890 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-15 18:04:15,086 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-15 18:04:15,129 INFO  [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-15 18:04:15,139 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-15 18:04:15,153 INFO  [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:15,919 INFO  [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-15 18:04:15,920 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-15 18:04:16,057 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> connection pointer in virConnectGetVersion
> 2021-12-15 18:04:16,066 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-15 18:04:16,066 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-15 18:04:16,151 INFO  [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-15 18:04:16,272 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-15 18:04:16,273 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-15 18:04:16,289 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-15 18:04:16,289 INFO  [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
>
> I also noticed this:
>
> 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle
> host connection:
> java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> hostOS: Rocky into a cluster,in which there are CentOS hosts added
> at
> com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
> at
> com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
> at
> com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
> at
> com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
> at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> at
> 

Re: Issues with libvirt after upgrading to RockyLinux 8.5

[Jeremy Hansen]

Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt and 
qemu and ca-certificates to the same version as the other functional nodes. 
That didn’t seem to help. This is obviously an ssl issue but I don’t really 
know what to do about it.

2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) (logid:) 
Agent started
2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) (logid:) 
Implementation Version is 4.15.0.0
2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) (logid:) 
agent.properties found at /etc/cloudstack/agent/agent.properties
2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) (logid:) 
Defaulting to using properties file for storage
2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) (logid:) 
Defaulting to the constant time backoff algorithm
2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j 
configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) (logid:) 
Using default Java settings for IPv6 preference for agent connection
2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0
2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) uefi properties file not found due to: Unable to find file 
uefi.properties.
2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null) 
(logid:) No existing libvirtd connection found. Opening a new one
2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) No libvirt.vif.driver specified. Defaults to 
BridgeVifDriver.
2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource] 
(main:null) (logid:) iscsi session clean up is disabled
2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id 
= 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host 
= 192.168.30.59 : port = 8250
2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:) 
Connecting to 192.168.30.59:8250
2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf file 
found: /etc/cloudstack/agent/agent.properties
2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: 
Handshake done
2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:) 
Connected to 192.168.30.59:8250
2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor] 
(Agent-Handler-1:null) (logid:) Attempting to create storage pool 
18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection] 
(Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid 
connection pointer in virConnectGetVersion
2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] 
(Agent-Handler-1:null) (logid:) Found existing defined storage pool 
18796842-a137-475d-9799-9874240e3c0c, using it.
2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] 
(Agent-Handler-1:null) (logid:) Trying to fetch storage pool 
18796842-a137-475d-9799-9874240e3c0c from libvirt
2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper] 
(Agent-Handler-1:null) (logid:) Default Builder inited.
2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null) 
(logid:) Proccess agent startup answer, agent id = 0
2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null) 
(logid:) Set agent id 0
2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null) 
(logid:) Startup Response Received: agent id = 0
2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (AgentShutdownThread:null) 
(logid:) Stopping the agent: Reason = sig.kill

I also noticed this:

2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle 
host connection:
java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with hostOS: 
Rocky into a cluster,in which there are CentOS hosts added
at 
com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
at 
com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
at 
com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
at 
com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at 

Re: Issue with Quick Start Guide

[Matthew Samani]

Daan,

Sorry for the delay. I am in school and I am juggling multiple classes.
Since I am having issues installing guest additions on VirtualBox, I tried
a workaround to get the wall of text from VirtualBox. It isn't perfect but
I corrected what I was able to find. SIDE NOTE: This is an error log from
today after trying a fresh install from scratch.

21-12-15 16:03:34,507 WARN [c.c.u.systemVmTemplateRegistration]
(main:nullt) (togid:) 4.16.0 LAC SystemVm template not Tound. Cannot upgrade
system Vms hypervisor is not used, so not failing upgrade

2021-12-15 16:03:34,808 WARN [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) 4.16.0 Hyperv SystemVm template not found. Cannot
upgrade
system Vms hypervisor is not used, so not failing upgrade

2021-12-15 16:03:34,810 WARN [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) 4.16.0 Ovm3 SystemVm template not found. Cannot upgrade
system Vms hypervisor is not used, so not failing upgrade

2021-12-15 16:03:34,811 WARN [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) 4.16.0 KVM SystemVm template not found. Cannot upgrade
system Vms hypervisor is not used, so not failing upgrade

2021-12-15 16:03:34,812 WARN [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) 4.16.0 VMware SystemVm template not found. Cannot
upgrade
system Vms hypervisor is not used, so not failing upgrade

2021-12-15 16:03:44,160 WARN [c.c.s.d.DownloadMonitorImpl] (main:null)
(Llogid:) Only realhostip.com ssl cert is supported, ignoring self-signed
and other certs

2021-12-15 16:03:44,186 DEBUG [o.a.c.u.m.SMTPMailSender] (main:null)
(logid:) Unable to instantiate SMTP mail session due to empty or null host
2021-12-15 16:03:44,233 DEBUG [o.a.c.u.m.SMTPMailsender] (main:null)
(logid:) Unable to instantiate SMTP mail session due to empty or null host
2021-12-15 16:03:44,954 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:)
Unable to find properties file: commands.properties
2021-12-15 16:03:45,415 DEBUG [o.a.c.u.m.SMTPMailsender] (main:null)
(logid:) Unable to instantiate SMTP mail session due to empty or null host
2021-12-15 16:03:49,759 INFO [c.c.h.x.r.XenServerConnectionPool]
(main:null) (logid:) XenServer Connection Pool Configs
sleep. interval.on.error=1
2021-12-15 16:03:50,037 WARN [c.c.c.ConfigurationManagerImpl]
(main:ctx-216caf38) (Logid:) Management network CIDR is not configured
originally
Set it default to 192.168.1.0/24

2021-12-15 16:03:50,039 WARN [c.c.a.AlertManagerImpl] (main:ctx-216caf38)
(logid:) alertType=[14] dataCenterId=[0] podId=[0] clusterId=[null]
message=[Management network CIDR is not configured originally. Set it
default to 192.168.1.0/24].

2021-12-15 16:03:50,049 WARN [c.c.a.AlertManagerImpl] (main:ctx-216caf38)
(logid:) No recipients set in ‘alert.email.addresses', skipping sending
alert with subject: Management network CIDR is not configured originally.
Set it default to 192.168.1.0/24 and content

2021-12-15 16:03:50,621 DEBUG [o.a.c.u.p.ProcessRunner] (main:ctx-216caf38)
(logid:) Process standard error output command [/usr/bin/ipmitool -V]:
1.

2021-12-15 16:03:52,899 WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-58de38df) (logid:873728ac) alertType=[14]
dataCenterId=[0]
podId=[0] clusterId=[null] message=[Management server node 192.168.1.2 is
up]

2021-12-15 16:03:52,902 WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-58de38df) (logid:873728ac) No recipients set in
‘alert.email.addresses', skipping sending alert with subject: Management
server node 192.168.1.2 is up and content

2021-12-15 16:03:52,904 WARN [c.c.c.ClusterManagerImpl]
(Cluster-Notification-1:ctx-58de38df) (logid:873728ac) Notifying management
server join
event took 5 ms

2021-12-15 16:08:49,989 WARN [c.c.r.ResourceLimitManagerImp1]
(ResourceCountChecker-1:ctx-33b5ae9a) (logid:6b9750f5) Discrepancy in the
resource
count (original count=@ correct count = 13) for type template for account
ID 1 is fixed during resource count recalculation

2021-12-15 16:26:10,492 WARN [c.c.a.d.ParamGenericValidationWorker]
(qtp2063763486-280:ctx-8bl7ae80 ctx-ec6e287b) (logid:de8lc60a) Received
lunknown parameters for command listZones. Unknown parameters : Listall

2021-12-15 16:26:12,618 WARN [c.c.a.ApiServer]
(qtp2063763486-23:ctx-d4c8c024 ctx-31d999d5) (logid:a791e160) Unknown API
command
‘ListAndSwitchSamlAccount,

2021-12-15 16:27:30,630 WARN [c.c.a.d.ParamGenericValidationWorker]
(qtp2063763486-19:ctx-bc650704 ctx-406876c5) (logid:e5356cfc) Received
lunknown parameters for command listHypervisors. Unknown parameters :
listall

2021-12-15 16:32:16,813 WARN [c.c.a.d.ParamGenericValidationWorker]
(qtp2063763486-19:ctx-737d961c ctx-c9e3f387) (logid:4d99103f) Received
unknown parameters for command addHost. Unknown parameters : clustertype

2021-12-15 16:42:40,336 WARN [c.c.a.d.ParamGenericValidationWorker]
(qtp2063763486-283:ctx-868ee423 ctx-746378f0) (logid:9e7aeb58) Received
unknown parameters for command listHostsMetrics. Unknown parameters :
listall

2021-12-15 

Re: Support for MySQL Galera

[Pieter Harvey]

Upon further testing I have some additional questions. Testing this on MySQL InnoDB Cluster 
(Group Replication) and as an alternative, Galera Cluster: To get CloudStack Management servers 
working with InnoDB Cluster I had to update the tables cloud.op_lock and cloud.op_nwgrp_work in 
create-schema.sql from using ENGINE=MEMORY to ENGINE=InnoDB. Does anyone forsee this being a 
problem with the inner workings of CloudStack, now and in future?I don't seem to get any errors 
"yet" and this does allow the use of multi-master semi-synchronous database 
clustering.-PieterOn 13 Dec 2021, at 13:10, Pieter Harvey 
 wrote:Hello,So doing some testing in order to run 
multiple management servers, and testing so far seems to fail. Is the 
cloudstack-setup-management command (and it's child scripts) compatible with MySQL 8.0 but in a 
multi-master Galera cluster (synchronous)? Anyone use galera on their management nodes?Patched 
MySQL obtained from galeracluster.com-Pieter

Re: Cloudstack Kubernetes Issue

[Mevludin Blazevic]

Hi,

is it also possible to use k8s version > 1.20 with ACS 4.15.2 (i.e. 
download the systemvm template 4.16.x and edit the kubernetes template 
config)? I am not able to get by KCS running when using k8s iso from 
v1.20, but k8s v1.16 works fine for ACS 4.15.2.


If not, up to which k8s version is the CoreOS template for KVM 
(http://dl.openvm.eu/cloudstack/coreos/x86_64/) compatible? Maybe I can 
create a k8s iso v1.19...


Regards,

Mevludin

Am 17.11.2021 um 09:12 schrieb Pearl d'Silva:

Hi,

Could you please share which version of Kubernetes you are trying to deploy. From 
ACS 4.16 onward, k8s versions > 1.20 are only supported.


Thanks,
Pearl


From: Seth Lyons 
Sent: Wednesday, November 17, 2021 2:51 AM
To: users@cloudstack.apache.org 
Subject: Cloudstack Kubernetes Issue

Hello,
I am running Cloudstacks 4.16 and trying to deploy the Kubernetes add on. When 
I add the service it is stuck in the Starting state, but I see that two 
instances are created and seem to be running when viewed through the console 
proxy.
I see this error in the logs which seems related:

2021-11-16 15:29:24,718 WARN  [c.c.k.c.a.KubernetesClusterActionWorker] 
(API-Job-Executor-12:ctx-aeed885c job-550 ctx-9066b68b) (logid:ac518757) Unable 
to retrieve VMs for Kubernetes cluster : Seth
2021-11-16 15:29:24,718 WARN  [c.c.k.c.a.KubernetesClusterActionWorker] 
(API-Job-Executor-12:ctx-aeed885c job-550 ctx-9066b68b) (logid:ac518757) Unable 
to retrieve control VM for Kubernetes cluster : Seth
2021-11-16 15:29:26,912 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-23:ctx-1628641a job-550/job-551 ctx-defc38ee) 
(logid:ac518757) STATS: Failed to alloc resource from host: 11 reservedCpu: 0, 
requested cpu: 4000, reservedMem: (0 bytes) 0, requested mem: (2.00 GB) 
2147483648
2021-11-16 15:29:34,893 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-24:ctx-efb62ae9 job-550/job-552 ctx-b435021d) 
(logid:ac518757) STATS: Failed to alloc resource from host: 11 reservedCpu: 0, 
requested cpu: 4000, reservedMem: (0 bytes) 0, requested mem: (2.00 GB) 
2147483648



[Photo]
[Logo]
Seth Lyons
DevOps Engineer | Automate io

[phone-icon]  (415) 757-4696
[phone-icon]  sly...@automateio.com
[phone-icon]  www.automateio.com
[Banner]



  




--
Mevludin Blazevic, M.Sc.

University of Koblenz-Landau
Computing Centre (GHRKO)
Universitaetsstrasse 1
D-56070 Koblenz, Germany
Room A023
Tel: +49 261/287-1326

Re: "Add LDAP account" returns empty user list

[Daan Hoogland]

mevludin,

the base dn should be just that, not any group below it. Did you try
clearing the search group principle?
If ldap.group.user.uniquemember is "uniquemember", the group should show
`uniquemember: uid=person1,ou=ou1,dc=my-domain, dc=de` for all those users,
and not member: `uid=person1,ou=ou1,dc=my-domain, dc=de`. It seems
something is off with your configuration in LDAP. I am not sure if this is
needed for autoimport, the the empty principle group would be if the
correct membership attribute isn't set.

On Tue, Dec 14, 2021 at 5:29 PM Mevludin Blazevic 
wrote:

> Hi Daan,
>
> value for ldap.group.user.uniquemember is "uniquemember". I have also
> tried to set up the basedn as "ou=ou1,dc=my-domain,dc=de" to get all
> users of ou1, list is still empty..
>
> Am 14.12.2021 um 16:55 schrieb Daan Hoogland:
> > ok Mevludin,
> > can try and you empty
> >
> > ldap.search.group.principle (remove the
> > "cn=cloustack-user,ou=Ou1,dc=my-domain,dc=de"), if you have one all your
> > users must have the memberOf attribute filled with that group.
> >
> >
> > Can you share your value for ldap.group.user.uniquemember?
> >
> >
> > On Tue, Dec 14, 2021 at 4:18 PM Mevludin Blazevic <
> mblaze...@uni-koblenz.de>
> > wrote:
> >
> >> Hi Daan,
> >>
> >> yes, I am trying to use the manual import, we will not have much
> >> Cloudstack users so manually importing them once would be enough.
> >>
> >> I've added the LDAP configuration via the GUI under Configuration ->
> >> LDAP Configuration (only server and port, no domain). Then I configured
> >> the basedn and the other properties from my previous e-mail using the
> >> Global Settings view.
> >>
> >> The users do not have a memberOf attribute yet. Nevertheless, the group
> >> knows its members and yes, the group has a series of uniqueMember
> >> attributes, for example:
> >>
> >> member: uid=person1,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person2,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person3,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person4,ou=ou1,dc=my-domain, dc=de
> >> member:
> >> member: uid=person5,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person6,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person7,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person8,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person9,ou=ou1,dc=my-domain, dc=de
> >> member: uid=person10,ou=ou1,dc=my-domain, dc=de
> >> memberUid: person1
> >> memberUid: person2
> >> memberUid: person3
> >> memberUid: person4
> >> memberUid: person5
> >> memberUid: person6
> >> memberUid: person7
> >> memberUid: person8
> >> memberUid: person9
> >> memberUid: person10
> >>
> >> Is the manual import possible if there is no memberOf attribute?
> >>
> >> Best Regards
> >>
> >> Mevludin
> >>
> >> Am 14.12.2021 um 12:36 schrieb Daan Hoogland:
> >>> Mevludin,
> >>> I suppose you are using the documentation to add your LDAP. which
> >> strategy
> >>> are you using, manual import, autoimport or autosync?
> >>> By the looks it seems you want the manual import, but I am not sure.
> >>> Does the user have a memberOf attribute?
> >>> Does the group cloudstack-user have a series of uniqueMember
> attributes?
> >>>
> >>>
> >>> On Tue, Dec 14, 2021 at 11:04 AM Mevludin Blazevic<
> >> mblaze...@uni-koblenz.de>
> >>> wrote:
> >>>
>  Hi all,
> 
>  when I try to set up a connection to our LDAP server I am getting an
>  empty list after clicking on the "Add LDAP button". I have already set
>  up the basedn, confuigured a bind.principal by using the dn (beginning
>  with uid= instead of cn=) and a bind password. No LDAP exception is
>  logged, but when I try to change the password or the principal dn I am
>  getting an LDAP exception, so I assume that the connection can be
>  established. My configuration:
> 
>  LDAP: my-ldap-server.de:389 (no domain was assigned)
>  basedn: dc=my-domain, dc=de
>  bind-principal: uid=,ou=ou1,dc=my-domain, dc=de
>  ldap.provider: openldap
>  ldap.group.object: groupOfUniqueNames
>  ldap.nested.groups.enable: true
>  ldap.search.group.principle: (for example
>  "cn=cloustack-user,ou=Ou1,dc=my-domain,dc=de")
>  ldap.user.memberof.attribute: memberOf
>  ldap.user.object: inetOrgPerson
>  ldap.username.attribute: uid
>  ldap.read.timeout: 1000
>  ldap.request.page.size: 1000
> 
>  For testing purposes, I run ldapsearch on the same machine where
>  cloudstack-management is installed. For example:
> 
>  ldapsearch -ZZ -LLL -o ldif-wrap=no -c -h my-ldap-server.de -D
>  "uid=,ou=,dc=my-domain, dc=de" -w "" -b
>  dc=my-domain, dc=de "(ou=ou1)" --> returning a (long) list of LDAP
> >> entrys
>  ldapsearch -ZZ -LLL -o ldif-wrap=no -c -h my-ldap-server.de -D
>  "uid=,ou=,dc=my-domain, dc=de" -w "" -b
>  dc=my-domain, dc=de "(cn=cloustack-user)" --> returning a dn with a
> list
>  of all group members
> 
>  ldapsearch -ZZ -LLL -o ldif-wrap=no -c -h