Attaching larger volumes
I’m trying to attach a larger, 1TB volume to my vm. Being served via rbd/ceph.
Cloudstack 4.18.1.0. The error I see, which isn’t very descriptive:
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]:
com.cloud.utils.exception.CloudRuntimeException: Unable to find suitable
primary storage when creating volume
[{"name":"gitlab-clx-corp-storage1","uuid":"fa1b5b41-cbfc-4cbc-a728-593ceea85c90"}].
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolume(VolumeOrchestrator.java:703)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeOnPrimaryStorage(VolumeOrchestrator.java:1140)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
com.sun.proxy.$Proxy227.createVolumeOnPrimaryStorage(Unknown Source)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:2242)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:4713)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4751)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Jan 28 03:53:55 dell1.cs.clx.corp java[2257551]:
Migrating secondary storage
I’m trying to migrate to new secondary storage. I’m receiving this error: Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: com.cloud.utils.exception.CloudRuntimeException: Complete migration failed as there are data objects which are not Ready - i.e, they may be in Migrating, creating, copying, etc. states Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.engine.orchestration.DataMigrationUtility.checkIfCompleteMigrationPossible(DataMigrationUtility.java:122) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.engine.orchestration.StorageOrchestrator.migrateData(StorageOrchestrator.java:149) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.storage.ImageStoreServiceImpl.migrateData(ImageStoreServiceImpl.java:157) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.sun.proxy.$Proxy386.migrateData(Unknown Source) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.api.command.admin.storage.MigrateSecondaryStorageDataCmd.execute(MigrateSecondaryStorageDataCmd.java:100) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) Jan 26 23:33:25 dell1.fr1.clx.corp java[1787564]: at
Re: Issues migrating primary storage
I’m trying to put my NFS primary storage in to maintenance mode, which I
believe is supposed to migrate all of its storage, correct? The problem is I
don’t know how to get a status on this job? I can’t really tell if it’s
working. Management server doesn’t really have anything in the logs…. I don’t
see any new images or images growing on the Ceph side. So I just don’t know if
it’s working or how far along the migration is.
-jeremy
> On Friday, Jan 19, 2024 at 12:34 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote:
> I’m still having issues. Is it unreasonable to migrate 1TB images over a 10G
> network? Any other ideas of things to try would be much appreciated.
>
> -jeremy
>
>
>
> > On Wednesday, Jan 17, 2024 at 12:49 PM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote:
> > Extending these timeouts in the “wait” configs seems to have helped. One of
> > my 1TB volumes is finally migrating.
> >
> > WHat’s I’ve noticed is if I allocate a new 1TB volume, I can migrate this
> > between NFS and Ceph and it takes only about a 1 minute. I assume this is
> > because it’s “thin provisioned” and there’s no actual data on the volume.
> >
> > But these other volumes I’m trying to move are also “thin provisioned” but
> > they’re a part of a LVM group. Does making a thin provisioned device part
> > of a LVM group defeat the thin provisioning? I know these volumes weren’t
> > full, but I thought perhaps there’s a chance that since it’s a pv in a LVM
> > config, that maybe that defeats the thin provisioning and it counts it as a
> > full 1TB volume? I’m just spitballing but I’m trying to understand how this
> > works so we can do the right thing when provisioning additional volumes.
> >
> > Also, the behavior I’m seeing is it takes a very long time before I see the
> > block image show up on the Ceph side. Perhaps it preallocated a image
> > before copying the data? But it seemed strange that I wouldn’t immidiately
> > see the image appear on the Ceph side after initiating a migration. It’s
> > hard to see what’s actually going on from the logs and the interface.
> >
> > Thanks
> > -jeremy
> >
> >
> >
> > > On Tuesday, Jan 16, 2024 at 11:29 PM, Jeremy Hansen > > (mailto:jer...@skidrow.la)> wrote:
> > > I changed copy.volume.wait to 72000
> > >
> > > But I just noticed:
> > >
> > > kvm.storage.online.migration.wait and kvm.storage.offline.migration.wait.
> > > Worth changing this?
> > >
> > > Thanks
> > > -jeremy
> > >
> > >
> > > > On Tuesday, Jan 16, 2024 at 11:01 PM, Jithin Raju
> > > > mailto:jithin.r...@shapeblue.com)> wrote:
> > > > Hi Jeremy,
> > > >
> > > > Have you checked the ‘wait’ parameter? Used as wait * 2 timeout.
> > > >
> > > > -Jithin
> > > >
> > > > From: Jeremy Hansen
> > > > Date: Wednesday, 17 January 2024 at 12:14 PM
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: Issues migrating primary storage
> > > > Unfortunately the upgrade didn’t help:
> > > >
> > > > Resource [StoragePool:3] is unreachable: Volume
> > > > [{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}]
> > > > migration failed due to
> > > > [com.cloud.utils.exception.CloudRuntimeException: Failed to copy
> > > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca
> > > > to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].
> > > >
> > > >
> > > >
> > > > Anything else I can try? I’m trying to move away from NFS completely.
> > > >
> > > > -jeremy
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> > > > mailto:sureshkumar.anapa...@gmail.com>>
> > > > wrote:
> > > > Hi Jeremy,
> > > >
> > > > Can you extend with the config 'migratewait' and check.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> > > >
> > > > wrote:
> > > >
> > > >
> > > > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD.
> > > > 1TB
> > > > volumes. These inevitably times out. I extended t
Re: Issues migrating primary storage
I’m still having issues. Is it unreasonable to migrate 1TB images over a 10G
network? Any other ideas of things to try would be much appreciated.
-jeremy
> On Wednesday, Jan 17, 2024 at 12:49 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote:
> Extending these timeouts in the “wait” configs seems to have helped. One of
> my 1TB volumes is finally migrating.
>
> WHat’s I’ve noticed is if I allocate a new 1TB volume, I can migrate this
> between NFS and Ceph and it takes only about a 1 minute. I assume this is
> because it’s “thin provisioned” and there’s no actual data on the volume.
>
> But these other volumes I’m trying to move are also “thin provisioned” but
> they’re a part of a LVM group. Does making a thin provisioned device part of
> a LVM group defeat the thin provisioning? I know these volumes weren’t full,
> but I thought perhaps there’s a chance that since it’s a pv in a LVM config,
> that maybe that defeats the thin provisioning and it counts it as a full 1TB
> volume? I’m just spitballing but I’m trying to understand how this works so
> we can do the right thing when provisioning additional volumes.
>
> Also, the behavior I’m seeing is it takes a very long time before I see the
> block image show up on the Ceph side. Perhaps it preallocated a image before
> copying the data? But it seemed strange that I wouldn’t immidiately see the
> image appear on the Ceph side after initiating a migration. It’s hard to see
> what’s actually going on from the logs and the interface.
>
> Thanks
> -jeremy
>
>
>
> > On Tuesday, Jan 16, 2024 at 11:29 PM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote:
> > I changed copy.volume.wait to 72000
> >
> > But I just noticed:
> >
> > kvm.storage.online.migration.wait and kvm.storage.offline.migration.wait.
> > Worth changing this?
> >
> > Thanks
> > -jeremy
> >
> >
> > > On Tuesday, Jan 16, 2024 at 11:01 PM, Jithin Raju
> > > mailto:jithin.r...@shapeblue.com)> wrote:
> > > Hi Jeremy,
> > >
> > > Have you checked the ‘wait’ parameter? Used as wait * 2 timeout.
> > >
> > > -Jithin
> > >
> > > From: Jeremy Hansen
> > > Date: Wednesday, 17 January 2024 at 12:14 PM
> > > To: users@cloudstack.apache.org
> > > Subject: Re: Issues migrating primary storage
> > > Unfortunately the upgrade didn’t help:
> > >
> > > Resource [StoragePool:3] is unreachable: Volume
> > > [{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}]
> > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca
> > > to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].
> > >
> > >
> > >
> > > Anything else I can try? I’m trying to move away from NFS completely.
> > >
> > > -jeremy
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> > > mailto:sureshkumar.anapa...@gmail.com>>
> > > wrote:
> > > Hi Jeremy,
> > >
> > > Can you extend with the config 'migratewait' and check.
> > >
> > > Regards,
> > > Suresh
> > >
> > > On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> > > wrote:
> > >
> > >
> > > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > > volumes. These inevitably times out. I extended these configs:
> > >
> > > copy.volume.wait=72000
> > > job.cancel.threshold.minutes=480
> > > job.expire.minutes=1440
> > >
> > > This helped with smaller volumes but large once still eventually fail.
> > >
> > > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > > to
> > > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > > }
> > >
> > > 2024-01-16 0
Re: Issues migrating primary storage
Extending these timeouts in the “wait” configs seems to have helped. One of my
1TB volumes is finally migrating.
WHat’s I’ve noticed is if I allocate a new 1TB volume, I can migrate this
between NFS and Ceph and it takes only about a 1 minute. I assume this is
because it’s “thin provisioned” and there’s no actual data on the volume.
But these other volumes I’m trying to move are also “thin provisioned” but
they’re a part of a LVM group. Does making a thin provisioned device part of a
LVM group defeat the thin provisioning? I know these volumes weren’t full, but
I thought perhaps there’s a chance that since it’s a pv in a LVM config, that
maybe that defeats the thin provisioning and it counts it as a full 1TB volume?
I’m just spitballing but I’m trying to understand how this works so we can do
the right thing when provisioning additional volumes.
Also, the behavior I’m seeing is it takes a very long time before I see the
block image show up on the Ceph side. Perhaps it preallocated a image before
copying the data? But it seemed strange that I wouldn’t immidiately see the
image appear on the Ceph side after initiating a migration. It’s hard to see
what’s actually going on from the logs and the interface.
Thanks
-jeremy
> On Tuesday, Jan 16, 2024 at 11:29 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote:
> I changed copy.volume.wait to 72000
>
> But I just noticed:
>
> kvm.storage.online.migration.wait and kvm.storage.offline.migration.wait.
> Worth changing this?
>
> Thanks
> -jeremy
>
>
> > On Tuesday, Jan 16, 2024 at 11:01 PM, Jithin Raju
> > mailto:jithin.r...@shapeblue.com)> wrote:
> > Hi Jeremy,
> >
> > Have you checked the ‘wait’ parameter? Used as wait * 2 timeout.
> >
> > -Jithin
> >
> > From: Jeremy Hansen
> > Date: Wednesday, 17 January 2024 at 12:14 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Issues migrating primary storage
> > Unfortunately the upgrade didn’t help:
> >
> > Resource [StoragePool:3] is unreachable: Volume
> > [{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}]
> > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca
> > to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].
> >
> >
> >
> > Anything else I can try? I’m trying to move away from NFS completely.
> >
> > -jeremy
> >
> >
> >
> >
> >
> >
> > On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> > mailto:sureshkumar.anapa...@gmail.com>>
> > wrote:
> > Hi Jeremy,
> >
> > Can you extend with the config 'migratewait' and check.
> >
> > Regards,
> > Suresh
> >
> > On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> > wrote:
> >
> >
> > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > volumes. These inevitably times out. I extended these configs:
> >
> > copy.volume.wait=72000
> > job.cancel.threshold.minutes=480
> > job.expire.minutes=1440
> >
> > This helped with smaller volumes but large once still eventually fail.
> >
> > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to
> > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > }
> >
> > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > (logid:d7d98b81) Failed to migrate volume
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > is unreachable: Volume
> > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > at
> > org.apache.cloudstack.engine.orchestration
Re: Issues migrating primary storage
I changed copy.volume.wait to 72000
But I just noticed:
kvm.storage.online.migration.wait and kvm.storage.offline.migration.wait. Worth
changing this?
Thanks
-jeremy
> On Tuesday, Jan 16, 2024 at 11:01 PM, Jithin Raju (mailto:jithin.r...@shapeblue.com)> wrote:
> Hi Jeremy,
>
> Have you checked the ‘wait’ parameter? Used as wait * 2 timeout.
>
> -Jithin
>
> From: Jeremy Hansen
> Date: Wednesday, 17 January 2024 at 12:14 PM
> To: users@cloudstack.apache.org
> Subject: Re: Issues migrating primary storage
> Unfortunately the upgrade didn’t help:
>
> Resource [StoragePool:3] is unreachable: Volume
> [{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}] migration
> failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to
> copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca
> to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].
>
>
>
> Anything else I can try? I’m trying to move away from NFS completely.
>
> -jeremy
>
>
>
>
>
>
> On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> mailto:sureshkumar.anapa...@gmail.com>> wrote:
> Hi Jeremy,
>
> Can you extend with the config 'migratewait' and check.
>
> Regards,
> Suresh
>
> On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> wrote:
>
>
> I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> volumes. These inevitably times out. I extended these configs:
>
> copy.volume.wait=72000
> job.cancel.threshold.minutes=480
> job.expire.minutes=1440
>
> This helped with smaller volumes but large once still eventually fail.
>
> 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> to
> b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> }
>
> 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> (logid:d7d98b81) Failed to migrate volume
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> is unreachable: Volume
> [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> Source)
> at
&
Re: Issues migrating primary storage
Unfortunately the upgrade didn’t help:
Resource [StoragePool:3] is unreachable: Volume
[{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}] migration
failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to copy
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca
to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].
Anything else I can try? I’m trying to move away from NFS completely.
-jeremy
> On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> mailto:sureshkumar.anapa...@gmail.com)>
> wrote:
> Hi Jeremy,
>
> Can you extend with the config 'migratewait' and check.
>
> Regards,
> Suresh
>
> On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> wrote:
>
> > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > volumes. These inevitably times out. I extended these configs:
> >
> > copy.volume.wait=72000
> > job.cancel.threshold.minutes=480
> > job.expire.minutes=1440
> >
> > This helped with smaller volumes but large once still eventually fail.
> >
> > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to
> > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > }
> >
> > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > (logid:d7d98b81) Failed to migrate volume
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > is unreachable: Volume
> > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > at
> > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> >
Re: Issues migrating primary storage
Upgraded to 4.18.1.0 and trying again…
-jeremy
> On Tuesday, Jan 16, 2024 at 7:08 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote:
> Unfortunately, this didn’t seem to have an impact. Volume migration still
> eventually fails. Should I move to 4.18.1.0?
>
> Thanks
> -jeremy
>
>
>
> > On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> > mailto:sureshkumar.anapa...@gmail.com)>
> > wrote:
> > Hi Jeremy,
> >
> > Can you extend with the config 'migratewait' and check.
> >
> > Regards,
> > Suresh
> >
> > On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> > wrote:
> >
> > > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > > volumes. These inevitably times out. I extended these configs:
> > >
> > > copy.volume.wait=72000
> > > job.cancel.threshold.minutes=480
> > > job.expire.minutes=1440
> > >
> > > This helped with smaller volumes but large once still eventually fail.
> > >
> > > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > > to
> > > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > > }
> > >
> > > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > > (logid:d7d98b81) Failed to migrate volume
> > > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > > is unreachable: Volume
> > > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > > at
> > > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> > > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> > > Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > at
> > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > > at
> > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> > > at
> > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> > > at
> > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> > > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> > > Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> > > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> > > Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
Re: Issues migrating primary storage
Unfortunately, this didn’t seem to have an impact. Volume migration still
eventually fails. Should I move to 4.18.1.0?
Thanks
-jeremy
> On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti
> mailto:sureshkumar.anapa...@gmail.com)>
> wrote:
> Hi Jeremy,
>
> Can you extend with the config 'migratewait' and check.
>
> Regards,
> Suresh
>
> On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen
> wrote:
>
> > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > volumes. These inevitably times out. I extended these configs:
> >
> > copy.volume.wait=72000
> > job.cancel.threshold.minutes=480
> > job.expire.minutes=1440
> >
> > This helped with smaller volumes but large once still eventually fail.
> >
> > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to
> > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > }
> >
> > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > (logid:d7d98b81) Failed to migrate volume
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > is unreachable: Volume
> > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > at
> > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex
Issues migrating primary storage
I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
volumes. These inevitably times out. I extended these configs:
copy.volume.wait=72000
job.cancel.threshold.minutes=480
job.expire.minutes=1440
This helped with smaller volumes but large once still eventually fail.
2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null)
(logid:) Seq 1-5583619113009291196: Processing: { Ans: , MgmtId:
20558852646968, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to copy
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
to
b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
(Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
(logid:d7d98b81) Failed to migrate volume
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is
unreachable: Volume
[{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
Failed to copy
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
I need to force delete primary storage definition
Is there a way I can delete a primary storage configuration if the storage no longer exists? This is a test cluster but I’d like to understand how to recover from this situation. The volumes that were left behind on this storage isn’t important. How can I force delete this storage or the two volumes left behind? 4.18.0.0. Thank you -jeremy signature.asc Description: PGP signature
Changing the primary storage name without disabling the storage pool?
I have a situation where I basically just want to change the primary storage endpoint name to a different hostname. The storage is all the same, but we’ve switched internal domain names. Doing a full migration really isn’t an option. It’s just the hostname. Is this something that can be done by direct database updates or does it go beyond the DB? The storage is RDB/Ceph. Cloudstack 4.18.1.0. Thanks -jeremy signature.asc Description: PGP signature
Re: Issue starting a specific VM
After some playing around, I’m able to get the instance up and running again,
but the issue still exists.
I noticed if I detect the two addition 100G DATADISKs, the instance starts. If
I reattach the volumes after the instance is running, everything is fine, but
as soon as I stop the instance and try to start again, I get the same failure.
Not sure how to find the root cause. Any ideas based on this information? These
two volumes are configured for lvm. I have many instance with a similar
configuration with no issues.
Thanks
-jeremy
> On Friday, Nov 17, 2023 at 1:06 AM, Me (mailto:jer...@skidrow.la)> wrote:
>
>
>
>
> > Begin forwarded message:
> >
> > From: Jeremy Hansen > (mailto:jer...@skidrow.la.INVALID)>
> > Date: Thursday, Nov 16, 2023 at 5:29 PM
> > To: mailto:users@cloudstack.apache.org)>
> > Subject: Issue starting a specific VM
> >
> > I noticed that. What’s odd is I made no changes to this VM.
> >
> > Is there a better way to list settings for a VM that isn’t running?
> >
> > Here are screenshots. Basically root and two addition volumes. One nic.
> >
> > Thanks!
> > -jeremy
> >
> >
> >
> > > On Thursday, Nov 16, 2023 at 4:56 PM, Wei ZHOU > > (mailto:ustcweiz...@gmail.com)> wrote:
> > > Hi,
> > >
> > > The logs show
> > >
> > > 2023-11-15 23:00:13,172 WARN [resource.wrapper.LibvirtStartCommandWrapper]
> > > (agentRequest-Handler-3:null) (logid:456eabd2) LibvirtException
> > > org.libvirt.LibvirtException: internal error: Only 1 IDE controller is
> > > supported
> > > at org.libvirt.ErrorHandler.processError(Unknown Source)
> > > at org.libvirt.ErrorHandler.processError(Unknown Source)
> > > at org.libvirt.Connect.domainCreateXML(Unknown Source)
> > > at
> > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1816)
> > >
> > > How many volumes/nics does the vm have ? Can you list the vm settings ?
> > >
> > >
> > > -Wei
> > >
> > >
> > > On Thu, 16 Nov 2023 at 08:06, Jeremy Hansen
> > > wrote:
> > >
> > > > I’m having an issue with a specific VM being unable to start and I’m not
> > > > quite sure why or how it got in this state.
> > > >
> > > > cloudstack 4.18.1.0
> > > >
> > > > Management log:
> > > >
> > > > 2023-11-15 22:56:44,938 ERROR [c.c.a.ApiAsyncJobDispatcher]
> > > > (API-Job-Executor-1:ctx-fc2bc1f9 job-78702) (logid:a3519842) Unexpected
> > > > exception while executing
> > > > org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM
> > > > [cc216a54-5d5b-4ec2-a178-474ff957e53f] due to [Unable to create a
> > > > deployment for VM instance
> > > > {"id":218,"instanceName":"i-4-218-VM","type":"User","uuid":"cc216a54-5d5b-4ec2-a178-474ff957e53f"}].
> > > > at
> > > > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:841)
> > > > at
> > > > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:258)
> > > > at
> > > > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:214)
> > > > at
> > > > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5401)
> > > > at
> > > > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5251)
> > > > at
> > > > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3179)
> > > > at
> > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > Method)
> > > > at
> > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > at
> > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > > > at
> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.in
Issue starting a specific VM
I’m having an issue with a specific VM being unable to start and I’m not quite
sure why or how it got in this state.
cloudstack 4.18.1.0
Management log:
2023-11-15 22:56:44,938 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-1:ctx-fc2bc1f9 job-78702) (logid:a3519842) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM
[cc216a54-5d5b-4ec2-a178-474ff957e53f] due to [Unable to create a deployment
for VM instance
{"id":218,"instanceName":"i-4-218-VM","type":"User","uuid":"cc216a54-5d5b-4ec2-a178-474ff957e53f"}].
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:841)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:258)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:214)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5401)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5251)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3179)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy185.startVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM instance
{"id":218,"instanceName":"i-4-218-VM","type":"User","uuid":"cc216a54-5d5b-4ec2-a178-474ff957e53f"}Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1226)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
... 18 more
Compute node logs:
==> agent.log <==
2023-11-15 23:00:10,017 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:5281116a) Trying to fetch storage pool
b221bcf2-819c-37c7-b007-bf7749ab468b from libvirt
2023-11-15 23:00:10,200 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:5281116a) Trying to fetch storage pool
Volume migration progress?
Is there anything I can do to verify a volume migration is still making progress? I’m doing a nfs -> ceph migration and it’s taking quiet a long time. 100G volume. I don’t really see anything useful in the management logs. 4.18.0.0. Thanks -jeremy signature.asc Description: PGP signature
Re: How do I login to Ubuntu’s cloud image?
So none of the UserData stuff seems to work for me. I cut and pasted this in to
the manual userdata area and I really can’t tell if it’s doing anything at all.
I’m setting for a user and supplying a password hash and I’m unable to log in
on the console.
I’m specifically using
https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img
and as far as I can tell, when it boots, network isn’t coming up because I see
nothing in DHCP that it’s asking for an ip address. As opposed to Rocky 9.1
qcow images which seem to work just fine out of the box. Network comes up, I
log in with my ssh key. No issue.
@Nux, I tried your image, but it didn’t boot. I see the
GRUB_FORCE_PARTUUID set, attempting initrdless boot.
error and it just halts.
I saw this when I was initially attempting to create the Ubuntu template.
Logically you would think the -kvm.img would indicate a qcow2 kvm image, but
this wouldn’t boot for me. The
https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img
seems to boot just fine.
Thanks
-jeremy
> On Monday, Mar 27, 2023 at 11:54 AM, Stephan Bienek (mailto:stephan@bienek.org)> wrote:
> Hi Jeremy,
>
> the last ubuntu cloud image i was using
> https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img
> was working with Cloudstack User-Data out of the box, without any
> modifications.
>
> Make sure to use
>
> Advanced mode / Show advanced settings / Userdata /( Manual Userdata entry /
> in case of cloudstack 4.18)
>
> and paste your cloud-init config, for example ("#cloud-config" line is
> required):
>
> #cloud-config
> users:
> - name: stephan
> ssh-authorized-keys:
> - ssh-ed25519
> lock_passwd: false
> passwd:
> sudo: ALL=(ALL) NOPASSWD:ALL
> shell: /bin/bash
> ssh_pwauth: true
>
> Are you really sure networking was not brought up?
> Make sure to use an Isolated network with a Virtual Router or any other
> network with "Config Drive" feature enabled.
> Networking is working without userdata - if this doesnt work you can give Nux
> idea a try to be 100% sure.
>
> Cheers,
> Stephan
>
>
> > Jeremy Hansen hat am 27.03.2023 18:13 CEST
> > geschrieben:
> >
> >
> > I’m trying to setup a template for Ubuntu 22.04LTS and while the image
> > boots successfully, I see no way to log in. It doesn’t seem to bring up
> > networking by default so I can’t ssh in with my pub key.
> >
> > Is this where “User Data” comes in to play? If so, anyone have an example
> > that sets the password or just brings up networking?
> >
> > Thanks!
> >
> >
> >
signature.asc
Description: PGP signature
How do I login to Ubuntu’s cloud image?
I’m trying to setup a template for Ubuntu 22.04LTS and while the image boots successfully, I see no way to log in. It doesn’t seem to bring up networking by default so I can’t ssh in with my pub key. Is this where “User Data” comes in to play? If so, anyone have an example that sets the password or just brings up networking? Thanks! signature.asc Description: PGP signature
Re: Rocky 9 management setup
I was able to fix this after restarting the management server after the initial install. I have 4.18 up and running now. Thanks -jeremy > On Friday, Mar 10, 2023 at 1:40 AM, Me (mailto:jer...@skidrow.la)> wrote: > So I went through the process on both 4.17 and 4.18 and made sure I chose > advanced and still, I see no way of adding a guest network in 4.18: > > > > 4.17 has a button to create a guest network. Do I have to change a parameter > in the global config to enable this? > > Thanks > -jeremy > > > > > > On Thursday, Mar 09, 2023 at 11:54 PM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi Jeremy, > > > > There is a new type of network: L2. Please try with the L2 network. > > refer to > > https://docs.cloudstack.apache.org/en/latest/adminguide/networking_and_traffic.html#configure-guest-traffic-in-an-advanced-zone > > > > Please note, L2 networks are only supported in the Advanced zone . > > If you created a Basic zone, you cannot create any new networks. > > > > > > Kind regards, > > Wei > > > > On Fri, 10 Mar 2023 at 06:22, Jeremy Hansen > > wrote: > > > > > Ran in to this while trying to setup QuickCloudNoServices: > > > > > > Configuring guest traffic > > > *Something went wrong; please correct the following:* > > > Service Dhcp is not supported by the network offering [Network Offering > > > [20-Guest-QuickCloudNoServices] > > > > > > This is during the configuration wizard. > > > > > > Not really sure I understand. I’m trying to set up a L2 network with an > > > external DHCP server. Is QuickCloudNoServices the wrong choice? > > > > > > -jeremy > > > > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 6:23 PM, Me wrote: > > > Is there links to any kind of updated documentation? I seem to be having > > > issues with libvirtd and setting up the agent on Rocky 9. I’m trying to > > > follow the 4.17 docs, but some things seem to no longer apply. > > > > > > Error 530 > > > Unable to add the host: Cannot find the server resources at > > > http://dell1.fr1.clx.corp > > > > > > > > > > > > Also noticed in the agent adding process, this being written to qemu.conf: > > > > > > #deprecation_behavior = "none"vnc_tls=1 > > > vnc_tls=1vnc_tls=1 > > > vnc_tls=1vnc_tls=1 > > > > > > which breaks libvirtd: > > > > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file > > > syntax error: /etc/libvirt/qemu.conf:955: expecting a separator > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU > > > state driver failed: configuration file syntax error: > > > /etc/libvirt/qemu.conf:955: expecting a separator > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state > > > initialization failed > > > > > > Fixing that config allowed libvirtd to start and I was able to add hosts. > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU > > > wrote: > > > Hi Jeremy, > > > > > > You can find the packages for 4.18.0.0-RC2 at > > > > > > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > > > The "el8" packages can also be used on rocky8/rocky9. > > > > > > -Wei > > > > > > > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > > > wrote: > > > > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > > packages are available, I’d love to try it. > > > > > > -jeremy > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > > rohit.ya...@shapeblue.com> wrote: > > > Hi Jeremy, > > > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > > recommended), wait until 4.18.0.0. > > > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > > > > > Regards. > > > > > > > > > From: Kiran Chavala > > > Sent: Thursday, March 9, 2023 14:14 > > > To: Vivek Kumar via users > > > Subject: Re: Rocky 9 management setup > > > > > > Hi Vivek > > > > > &g
Re: Rocky 9 management setup
Seeing some broken labels: Also noticed that /etc/sysconfig/libvirtd needs to exist. The latest libvirt packages in Rocky 9 do not automatically place this file: From /var/log/cloudstack/agent/setup.log DEBUG:root:[Errno 2] No such file or directory: '/etc/sysconfig/libvirtd' File "/usr/lib64/python3.9/site-packages/cloudutils/serviceConfig.py", line 38, in configuration result = self.config() File "/usr/lib64/python3.9/site-packages/cloudutils/serviceConfig.py", line 635, in config cfo.save() File "/usr/lib64/python3.9/site-packages/cloudutils/configFileOps.py", line 62, in save fp = open(self.fileName, "r") -jeremy > On Friday, Mar 10, 2023 at 1:40 AM, Me (mailto:jer...@skidrow.la)> wrote: > So I went through the process on both 4.17 and 4.18 and made sure I chose > advanced and still, I see no way of adding a guest network in 4.18: > > > > 4.17 has a button to create a guest network. Do I have to change a parameter > in the global config to enable this? > > Thanks > -jeremy > > > > > > On Thursday, Mar 09, 2023 at 11:54 PM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi Jeremy, > > > > There is a new type of network: L2. Please try with the L2 network. > > refer to > > https://docs.cloudstack.apache.org/en/latest/adminguide/networking_and_traffic.html#configure-guest-traffic-in-an-advanced-zone > > > > Please note, L2 networks are only supported in the Advanced zone . > > If you created a Basic zone, you cannot create any new networks. > > > > > > Kind regards, > > Wei > > > > On Fri, 10 Mar 2023 at 06:22, Jeremy Hansen > > wrote: > > > > > Ran in to this while trying to setup QuickCloudNoServices: > > > > > > Configuring guest traffic > > > *Something went wrong; please correct the following:* > > > Service Dhcp is not supported by the network offering [Network Offering > > > [20-Guest-QuickCloudNoServices] > > > > > > This is during the configuration wizard. > > > > > > Not really sure I understand. I’m trying to set up a L2 network with an > > > external DHCP server. Is QuickCloudNoServices the wrong choice? > > > > > > -jeremy > > > > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 6:23 PM, Me wrote: > > > Is there links to any kind of updated documentation? I seem to be having > > > issues with libvirtd and setting up the agent on Rocky 9. I’m trying to > > > follow the 4.17 docs, but some things seem to no longer apply. > > > > > > Error 530 > > > Unable to add the host: Cannot find the server resources at > > > http://dell1.fr1.clx.corp > > > > > > > > > > > > Also noticed in the agent adding process, this being written to qemu.conf: > > > > > > #deprecation_behavior = "none"vnc_tls=1 > > > vnc_tls=1vnc_tls=1 > > > vnc_tls=1vnc_tls=1 > > > > > > which breaks libvirtd: > > > > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file > > > syntax error: /etc/libvirt/qemu.conf:955: expecting a separator > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU > > > state driver failed: configuration file syntax error: > > > /etc/libvirt/qemu.conf:955: expecting a separator > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state > > > initialization failed > > > > > > Fixing that config allowed libvirtd to start and I was able to add hosts. > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU > > > wrote: > > > Hi Jeremy, > > > > > > You can find the packages for 4.18.0.0-RC2 at > > > > > > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > > > The "el8" packages can also be used on rocky8/rocky9. > > > > > > -Wei > > > > > > > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > > > wrote: > > > > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > > packages are available, I’d love to try it. > > > > > > -jeremy > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > > rohit.ya...@shapeblue.com> wrote: > > > Hi Jeremy, > > > > > > EL9 isn't supported for 4.17 yet (you
Re: Rocky 9 management setup
So I went through the process on both 4.17 and 4.18 and made sure I chose advanced and still, I see no way of adding a guest network in 4.18: 4.17 has a button to create a guest network. Do I have to change a parameter in the global config to enable this? Thanks -jeremy > On Thursday, Mar 09, 2023 at 11:54 PM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > There is a new type of network: L2. Please try with the L2 network. > refer to > https://docs.cloudstack.apache.org/en/latest/adminguide/networking_and_traffic.html#configure-guest-traffic-in-an-advanced-zone > > Please note, L2 networks are only supported in the Advanced zone . > If you created a Basic zone, you cannot create any new networks. > > > Kind regards, > Wei > > On Fri, 10 Mar 2023 at 06:22, Jeremy Hansen > wrote: > > > Ran in to this while trying to setup QuickCloudNoServices: > > > > Configuring guest traffic > > *Something went wrong; please correct the following:* > > Service Dhcp is not supported by the network offering [Network Offering > > [20-Guest-QuickCloudNoServices] > > > > This is during the configuration wizard. > > > > Not really sure I understand. I’m trying to set up a L2 network with an > > external DHCP server. Is QuickCloudNoServices the wrong choice? > > > > -jeremy > > > > > > > > > > On Thursday, Mar 09, 2023 at 6:23 PM, Me wrote: > > Is there links to any kind of updated documentation? I seem to be having > > issues with libvirtd and setting up the agent on Rocky 9. I’m trying to > > follow the 4.17 docs, but some things seem to no longer apply. > > > > Error 530 > > Unable to add the host: Cannot find the server resources at > > http://dell1.fr1.clx.corp > > > > > > > > Also noticed in the agent adding process, this being written to qemu.conf: > > > > #deprecation_behavior = "none"vnc_tls=1 > > vnc_tls=1vnc_tls=1 > > vnc_tls=1vnc_tls=1 > > > > which breaks libvirtd: > > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file > > syntax error: /etc/libvirt/qemu.conf:955: expecting a separator > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU > > state driver failed: configuration file syntax error: > > /etc/libvirt/qemu.conf:955: expecting a separator > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state > > initialization failed > > > > Fixing that config allowed libvirtd to start and I was able to add hosts. > > > > Thanks > > -jeremy > > > > > > > > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU > > wrote: > > Hi Jeremy, > > > > You can find the packages for 4.18.0.0-RC2 at > > > > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > > The "el8" packages can also be used on rocky8/rocky9. > > > > -Wei > > > > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > > wrote: > > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > packages are available, I’d love to try it. > > > > -jeremy > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > rohit.ya...@shapeblue.com> wrote: > > Hi Jeremy, > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > recommended), wait until 4.18.0.0. > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > Regards. > > > > > > From: Kiran Chavala > > Sent: Thursday, March 9, 2023 14:14 > > To: Vivek Kumar via users > > Subject: Re: Rocky 9 management setup > > > > Hi Vivek > > > > You can comment the line number 45 and line 375 to 391 in the script > > /usr/bin/cloudstack-setup-databases > > > > And try running the cloudstack-setup-databases again > > > > Regards > > Kiran > > > > > > > > > > > > > > > > > > > > From: Jeremy Hansen > > Sent: Thursday, March 09, 2023 07:53 > > To: Vivek Kumar via users > > Subject: Rocky 9 management setup > > > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize > > Rocky isn’t officially supported yet but I’d like to try… > > > > Is there an alternative way to setup the db on a fresh install? > > > > Currently seeing > > > > Traceback (most recent call last): > > File "/usr/bin/cloudstack-setup-databases", line 45, in > > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > > > Do the latest builds support Rocky 9? > > > > Thanks > > -jeremy > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Rocky 9 management setup
Got it. I’ll try that. Thank you. -jeremy > On Thursday, Mar 09, 2023 at 11:54 PM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > There is a new type of network: L2. Please try with the L2 network. > refer to > https://docs.cloudstack.apache.org/en/latest/adminguide/networking_and_traffic.html#configure-guest-traffic-in-an-advanced-zone > > Please note, L2 networks are only supported in the Advanced zone . > If you created a Basic zone, you cannot create any new networks. > > > Kind regards, > Wei > > On Fri, 10 Mar 2023 at 06:22, Jeremy Hansen > wrote: > > > Ran in to this while trying to setup QuickCloudNoServices: > > > > Configuring guest traffic > > *Something went wrong; please correct the following:* > > Service Dhcp is not supported by the network offering [Network Offering > > [20-Guest-QuickCloudNoServices] > > > > This is during the configuration wizard. > > > > Not really sure I understand. I’m trying to set up a L2 network with an > > external DHCP server. Is QuickCloudNoServices the wrong choice? > > > > -jeremy > > > > > > > > > > On Thursday, Mar 09, 2023 at 6:23 PM, Me wrote: > > Is there links to any kind of updated documentation? I seem to be having > > issues with libvirtd and setting up the agent on Rocky 9. I’m trying to > > follow the 4.17 docs, but some things seem to no longer apply. > > > > Error 530 > > Unable to add the host: Cannot find the server resources at > > http://dell1.fr1.clx.corp > > > > > > > > Also noticed in the agent adding process, this being written to qemu.conf: > > > > #deprecation_behavior = "none"vnc_tls=1 > > vnc_tls=1vnc_tls=1 > > vnc_tls=1vnc_tls=1 > > > > which breaks libvirtd: > > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file > > syntax error: /etc/libvirt/qemu.conf:955: expecting a separator > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU > > state driver failed: configuration file syntax error: > > /etc/libvirt/qemu.conf:955: expecting a separator > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state > > initialization failed > > > > Fixing that config allowed libvirtd to start and I was able to add hosts. > > > > Thanks > > -jeremy > > > > > > > > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU > > wrote: > > Hi Jeremy, > > > > You can find the packages for 4.18.0.0-RC2 at > > > > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > > The "el8" packages can also be used on rocky8/rocky9. > > > > -Wei > > > > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > > wrote: > > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > packages are available, I’d love to try it. > > > > -jeremy > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > rohit.ya...@shapeblue.com> wrote: > > Hi Jeremy, > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > recommended), wait until 4.18.0.0. > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > Regards. > > > > > > From: Kiran Chavala > > Sent: Thursday, March 9, 2023 14:14 > > To: Vivek Kumar via users > > Subject: Re: Rocky 9 management setup > > > > Hi Vivek > > > > You can comment the line number 45 and line 375 to 391 in the script > > /usr/bin/cloudstack-setup-databases > > > > And try running the cloudstack-setup-databases again > > > > Regards > > Kiran > > > > > > > > > > > > > > > > > > > > From: Jeremy Hansen > > Sent: Thursday, March 09, 2023 07:53 > > To: Vivek Kumar via users > > Subject: Rocky 9 management setup > > > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize > > Rocky isn’t officially supported yet but I’d like to try… > > > > Is there an alternative way to setup the db on a fresh install? > > > > Currently seeing > > > > Traceback (most recent call last): > > File "/usr/bin/cloudstack-setup-databases", line 45, in > > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > > > Do the latest builds support Rocky 9? > > > > Thanks > > -jeremy > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Rocky 9 management setup
I don’t see a way to add a guest network after the setup wizard. Not sure if that changed. -jeremy > On Thursday, Mar 09, 2023 at 9:22 PM, Me (mailto:jer...@skidrow.la)> wrote: > Ran in to this while trying to setup QuickCloudNoServices: > > Configuring guest traffic > Something went wrong; please correct the following: > Service Dhcp is not supported by the network offering [Network Offering > [20-Guest-QuickCloudNoServices] > > This is during the configuration wizard. > > Not really sure I understand. I’m trying to set up a L2 network with an > external DHCP server. Is QuickCloudNoServices the wrong choice? > > -jeremy > > > > > > On Thursday, Mar 09, 2023 at 6:23 PM, Me > (mailto:jer...@skidrow.la)> wrote: > > Is there links to any kind of updated documentation? I seem to be having > > issues with libvirtd and setting up the agent on Rocky 9. I’m trying to > > follow the 4.17 docs, but some things seem to no longer apply. > > > > Error 530 > > Unable to add the host: Cannot find the server resources at > > http://dell1.fr1.clx.corp > > > > > > > > > > Also noticed in the agent adding process, this being written to qemu.conf: > > > > #deprecation_behavior = "none"vnc_tls=1 > > vnc_tls=1vnc_tls=1 > > vnc_tls=1vnc_tls=1 > > > > > > which breaks libvirtd: > > > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file > > syntax error: /etc/libvirt/qemu.conf:955: expecting a separator > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU > > state driver failed: configuration file syntax error: > > /etc/libvirt/qemu.conf:955: expecting a separator > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state > > initialization failed > > > > > > Fixing that config allowed libvirtd to start and I was able to add hosts. > > > > Thanks > > -jeremy > > > > > > > > > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU > > (mailto:ustcweiz...@gmail.com)> wrote: > > > Hi Jeremy, > > > > > > You can find the packages for 4.18.0.0-RC2 at > > > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > > > The "el8" packages can also be used on rocky8/rocky9. > > > > > > -Wei > > > > > > > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > > > wrote: > > > > > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > > > packages are available, I’d love to try it. > > > > > > > > -jeremy > > > > > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > > > rohit.ya...@shapeblue.com> wrote: > > > > Hi Jeremy, > > > > > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > > > recommended), wait until 4.18.0.0. > > > > > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > > > > > > > > > Regards. > > > > > > > > > > > > From: Kiran Chavala > > > > Sent: Thursday, March 9, 2023 14:14 > > > > To: Vivek Kumar via users > > > > Subject: Re: Rocky 9 management setup > > > > > > > > Hi Vivek > > > > > > > > You can comment the line number 45 and line 375 to 391 in the script > > > > /usr/bin/cloudstack-setup-databases > > > > > > > > And try running the cloudstack-setup-databases again > > > > > > > > Regards > > > > Kiran > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Jeremy Hansen > > > > Sent: Thursday, March 09, 2023 07:53 > > > > To: Vivek Kumar via users > > > > Subject: Rocky 9 management setup > > > > > > > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I > > > > realize > > > > Rocky isn’t officially supported yet but I’d like to try… > > > > > > > > Is there an alternative way to setup the db on a fresh install? > > > > > > > > Currently seeing > > > > > > > > Traceback (most recent call last): > > > > File "/usr/bin/cloudstack-setup-databases", line 45, in > > > > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > > > > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > > > > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > > > > > > > Do the latest builds support Rocky 9? > > > > > > > > Thanks > > > > -jeremy > > > > > > > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Rocky 9 management setup
Ran in to this while trying to setup QuickCloudNoServices: Configuring guest traffic Something went wrong; please correct the following: Service Dhcp is not supported by the network offering [Network Offering [20-Guest-QuickCloudNoServices] This is during the configuration wizard. Not really sure I understand. I’m trying to set up a L2 network with an external DHCP server. Is QuickCloudNoServices the wrong choice? -jeremy > On Thursday, Mar 09, 2023 at 6:23 PM, Me (mailto:jer...@skidrow.la)> wrote: > Is there links to any kind of updated documentation? I seem to be having > issues with libvirtd and setting up the agent on Rocky 9. I’m trying to > follow the 4.17 docs, but some things seem to no longer apply. > > Error 530 > Unable to add the host: Cannot find the server resources at > http://dell1.fr1.clx.corp > > > > > Also noticed in the agent adding process, this being written to qemu.conf: > > #deprecation_behavior = "none"vnc_tls=1 > vnc_tls=1vnc_tls=1 > vnc_tls=1vnc_tls=1 > > > which breaks libvirtd: > > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file syntax > error: /etc/libvirt/qemu.conf:955: expecting a separator > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU > state driver failed: configuration file syntax error: > /etc/libvirt/qemu.conf:955: expecting a separator > Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state > initialization failed > > > Fixing that config allowed libvirtd to start and I was able to add hosts. > > Thanks > -jeremy > > > > > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi Jeremy, > > > > You can find the packages for 4.18.0.0-RC2 at > > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > > The "el8" packages can also be used on rocky8/rocky9. > > > > -Wei > > > > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > > wrote: > > > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > > packages are available, I’d love to try it. > > > > > > -jeremy > > > > > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > > rohit.ya...@shapeblue.com> wrote: > > > Hi Jeremy, > > > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > > recommended), wait until 4.18.0.0. > > > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > > > > > Regards. > > > > > > > > > From: Kiran Chavala > > > Sent: Thursday, March 9, 2023 14:14 > > > To: Vivek Kumar via users > > > Subject: Re: Rocky 9 management setup > > > > > > Hi Vivek > > > > > > You can comment the line number 45 and line 375 to 391 in the script > > > /usr/bin/cloudstack-setup-databases > > > > > > And try running the cloudstack-setup-databases again > > > > > > Regards > > > Kiran > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Jeremy Hansen > > > Sent: Thursday, March 09, 2023 07:53 > > > To: Vivek Kumar via users > > > Subject: Rocky 9 management setup > > > > > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize > > > Rocky isn’t officially supported yet but I’d like to try… > > > > > > Is there an alternative way to setup the db on a fresh install? > > > > > > Currently seeing > > > > > > Traceback (most recent call last): > > > File "/usr/bin/cloudstack-setup-databases", line 45, in > > > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > > > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > > > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > > > > > Do the latest builds support Rocky 9? > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Rocky 9 management setup
Is there links to any kind of updated documentation? I seem to be having issues with libvirtd and setting up the agent on Rocky 9. I’m trying to follow the 4.17 docs, but some things seem to no longer apply. Error 530 Unable to add the host: Cannot find the server resources at http://dell1.fr1.clx.corp Also noticed in the agent adding process, this being written to qemu.conf: #deprecation_behavior = "none"vnc_tls=1 vnc_tls=1vnc_tls=1 vnc_tls=1vnc_tls=1 which breaks libvirtd: Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: configuration file syntax error: /etc/libvirt/qemu.conf:955: expecting a separator Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Initialization of QEMU state driver failed: configuration file syntax error: /etc/libvirt/qemu.conf:955: expecting a separator Mar 09 18:14:21 dell1.fr1.clx.corp libvirtd[29917]: Driver state initialization failed Fixing that config allowed libvirtd to start and I was able to add hosts. Thanks -jeremy > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > You can find the packages for 4.18.0.0-RC2 at > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > The "el8" packages can also be used on rocky8/rocky9. > > -Wei > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > wrote: > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > packages are available, I’d love to try it. > > > > -jeremy > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > rohit.ya...@shapeblue.com> wrote: > > Hi Jeremy, > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > recommended), wait until 4.18.0.0. > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > Regards. > > > > > > From: Kiran Chavala > > Sent: Thursday, March 9, 2023 14:14 > > To: Vivek Kumar via users > > Subject: Re: Rocky 9 management setup > > > > Hi Vivek > > > > You can comment the line number 45 and line 375 to 391 in the script > > /usr/bin/cloudstack-setup-databases > > > > And try running the cloudstack-setup-databases again > > > > Regards > > Kiran > > > > > > > > > > > > > > > > > > > > From: Jeremy Hansen > > Sent: Thursday, March 09, 2023 07:53 > > To: Vivek Kumar via users > > Subject: Rocky 9 management setup > > > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize > > Rocky isn’t officially supported yet but I’d like to try… > > > > Is there an alternative way to setup the db on a fresh install? > > > > Currently seeing > > > > Traceback (most recent call last): > > File "/usr/bin/cloudstack-setup-databases", line 45, in > > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > > > Do the latest builds support Rocky 9? > > > > Thanks > > -jeremy > > > > > > > > > > signature.asc Description: PGP signature
Re: Rocky 9 management setup
Thank you! > On Thursday, Mar 09, 2023 at 10:22 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > You can find the packages for 4.18.0.0-RC2 at > http://packages.shapeblue.com/cloudstack/upstream/testing/4.18.0.0-RC20230224T1301/ > The "el8" packages can also be used on rocky8/rocky9. > > -Wei > > > On Thu, 9 Mar 2023 at 18:06, Jeremy Hansen > wrote: > > > Thanks. I had build issues with 4.18 trying to build on Rocky 9. If > > packages are available, I’d love to try it. > > > > -jeremy > > > > > > > > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav < > > rohit.ya...@shapeblue.com> wrote: > > Hi Jeremy, > > > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > > recommended), wait until 4.18.0.0. > > > > You may even help test the 4.18.0.0 RCs. > > > > > > > > Regards. > > > > > > From: Kiran Chavala > > Sent: Thursday, March 9, 2023 14:14 > > To: Vivek Kumar via users > > Subject: Re: Rocky 9 management setup > > > > Hi Vivek > > > > You can comment the line number 45 and line 375 to 391 in the script > > /usr/bin/cloudstack-setup-databases > > > > And try running the cloudstack-setup-databases again > > > > Regards > > Kiran > > > > > > > > > > > > > > > > > > > > From: Jeremy Hansen > > Sent: Thursday, March 09, 2023 07:53 > > To: Vivek Kumar via users > > Subject: Rocky 9 management setup > > > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize > > Rocky isn’t officially supported yet but I’d like to try… > > > > Is there an alternative way to setup the db on a fresh install? > > > > Currently seeing > > > > Traceback (most recent call last): > > File "/usr/bin/cloudstack-setup-databases", line 45, in > > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > > > Do the latest builds support Rocky 9? > > > > Thanks > > -jeremy > > > > > > > > > > signature.asc Description: PGP signature
Re: Rocky 9 management setup
Thanks. I had build issues with 4.18 trying to build on Rocky 9. If packages are available, I’d love to try it. -jeremy > On Thursday, Mar 09, 2023 at 2:46 AM, Rohit Yadav (mailto:rohit.ya...@shapeblue.com)> wrote: > Hi Jeremy, > > EL9 isn't supported for 4.17 yet (you may workaround, but not generally > recommended), wait until 4.18.0.0. > > You may even help test the 4.18.0.0 RCs. > > > > Regards. > > > From: Kiran Chavala > Sent: Thursday, March 9, 2023 14:14 > To: Vivek Kumar via users > Subject: Re: Rocky 9 management setup > > Hi Vivek > > You can comment the line number 45 and line 375 to 391 in the script > /usr/bin/cloudstack-setup-databases > > And try running the cloudstack-setup-databases again > > Regards > Kiran > > > > > > > > > > From: Jeremy Hansen > Sent: Thursday, March 09, 2023 07:53 > To: Vivek Kumar via users > Subject: Rocky 9 management setup > > The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize > Rocky isn’t officially supported yet but I’d like to try… > > Is there an alternative way to setup the db on a fresh install? > > Currently seeing > > Traceback (most recent call last): > File "/usr/bin/cloudstack-setup-databases", line 45, in > from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 > ImportError: cannot import name 'check_selinux' from 'cloud_utils' > (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) > > Do the latest builds support Rocky 9? > > Thanks > -jeremy > > > > signature.asc Description: PGP signature
Rocky 9 management setup
The database setup script seems broken in 4.17.2.0 for Rocky 9. I realize Rocky isn’t officially supported yet but I’d like to try… Is there an alternative way to setup the db on a fresh install? Currently seeing Traceback (most recent call last): File "/usr/bin/cloudstack-setup-databases", line 45, in from cloud_utils import check_selinux, CheckFailed, resolves_to_ipv6 ImportError: cannot import name 'check_selinux' from 'cloud_utils' (/usr/local/lib/python3.9/site-packages/cloud_utils/__init__.py) Do the latest builds support Rocky 9? Thanks -jeremy signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
No issue with writes: 192.168.210.23:/exports/cloudstorage/primary 49T 57G 47T 1% /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020 tmpfs 6.3G 0 6.3G 0% /run/user/0 192.168.210.23:/exports/cloudstorage/secondary 49T 57G 47T 1% /var/cloudstack/mnt/161333239336.2b9f6261 [root@droid 11cd19d0-f207-3d01-880f-8d01d4b15020]# touch /var/cloudstack/mnt/161333239336.2b9f6261/file [root@droid 11cd19d0-f207-3d01-880f-8d01d4b15020]# ls -lad /var/cloudstack/mnt/161333239336.2b9f6261/file -rw-r--r-- 1 root root 0 Feb 22 17:30 /var/cloudstack/mnt/161333239336.2b9f6261/file [root@droid ~]# touch /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file [root@droid ~]# ls -ald /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file -rw-r--r-- 1 root root 0 Feb 22 17:31 /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file -jeremy > On Wednesday, Feb 22, 2023 at 5:07 AM, Simon Weller (mailto:siwelle...@gmail.com)> wrote: > Jeremy, > > Any chance you have a write permission problem on your new NFS server? > Those errors indicate an underlying storage issue. > > -Si > > On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen > wrote: > > > Oh and the system vm’s continue to stay in Starting state. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 9:44 PM, Me wrote: > > The vm’s finally stopped and restarted. This is what I’m seeing in dmesg > > on the secondary storage vm: > > > > root@s-60-VM:~# dmesg | grep -i error > > [ 3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 3.869553] systemd[1]: Failed to read configured hostname: > > Input/output error > > [ 4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro > > [ 4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 10.415215] print_req_error: 16 callbacks suppressed > > [ 10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 8:57 PM, Me wrote: > > The node cloudstack is claiming the system vm’s is starting on shows no > > signs of any vm’s running. virsh list is black. > > > > Thanks > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 8:23 PM, Me wrote: > > Also, just to note, I’m not sure how much made it in to the logs. The > > system vm’s are stuck in starting state and trying to kill through the > > interface doesn’t seem to do anything. > > > > -jeremy > > > > > > > > > > On Tuesday, Feb 21, 2023 at 8:20 PM, Me wrote: > > Is there something else I can use to submit logs? Too much for pastebin. > > > > Thanks > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon
Re: Stuck in Preparing for maintenance on primary storage
Oh and the system vm’s continue to stay in Starting state. -jeremy > On Tuesday, Feb 21, 2023 at 9:44 PM, Me (mailto:jer...@skidrow.la)> wrote: > The vm’s finally stopped and restarted. This is what I’m seeing in dmesg on > the secondary storage vm: > > root@s-60-VM:~# dmesg | grep -i error > [ 3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 3.869553] systemd[1]: Failed to read configured hostname: Input/output error > [ 4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro > [ 4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 10.415215] print_req_error: 16 callbacks suppressed > [ 10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > > > -jeremy > > > > > On Tuesday, Feb 21, 2023 at 8:57 PM, Me > (mailto:jer...@skidrow.la)> wrote: > > The node cloudstack is claiming the system vm’s is starting on shows no > > signs of any vm’s running. virsh list is black. > > > > Thanks > > -jeremy > > > > > > > > > On Tuesday, Feb 21, 2023 at 8:23 PM, Me > > (mailto:jer...@skidrow.la)> wrote: > > > Also, just to note, I’m not sure how much made it in to the logs. The > > > system vm’s are stuck in starting state and trying to kill through the > > > interface doesn’t seem to do anything. > > > > > > -jeremy > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 8:20 PM, Me > > > (mailto:jer...@skidrow.la)> wrote: > > > > Is there something else I can use to submit logs? Too much for pastebin. > > > > > > > > Thanks > > > > -jeremy > > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller > > > > > mailto:siwelle...@gmail.com)> wrote: > > > > > Can you pull some management server logs and also put the CloudStack > > > > > KVM > > > > > agent into debug mode before destroying the ssvm and share the logs? > > > > > > > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > > > > > > > > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen > > > > > > > > > > wrote: > > > > > > > > > > > Yes. It’s just a different partition on the same nfs server. > > > > > > > > > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller > > > > > > > > > > > > wrote: > > > > > > The new and old primary storage is in the same zone, correct? > > > > > > Did you al
Re: Stuck in Preparing for maintenance on primary storage
The vm’s finally stopped and restarted. This is what I’m seeing in dmesg on the secondary storage vm: root@s-60-VM:~# dmesg | grep -i error [ 3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 3.869553] systemd[1]: Failed to read configured hostname: Input/output error [ 4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro [ 4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 10.415215] print_req_error: 16 callbacks suppressed [ 10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 -jeremy > On Tuesday, Feb 21, 2023 at 8:57 PM, Me (mailto:jer...@skidrow.la)> wrote: > The node cloudstack is claiming the system vm’s is starting on shows no signs > of any vm’s running. virsh list is black. > > Thanks > -jeremy > > > > > On Tuesday, Feb 21, 2023 at 8:23 PM, Me > (mailto:jer...@skidrow.la)> wrote: > > Also, just to note, I’m not sure how much made it in to the logs. The > > system vm’s are stuck in starting state and trying to kill through the > > interface doesn’t seem to do anything. > > > > -jeremy > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 8:20 PM, Me > > (mailto:jer...@skidrow.la)> wrote: > > > Is there something else I can use to submit logs? Too much for pastebin. > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller > > > (mailto:siwelle...@gmail.com)> wrote: > > > > Can you pull some management server logs and also put the CloudStack KVM > > > > agent into debug mode before destroying the ssvm and share the logs? > > > > > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > > > > > > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen > > > > wrote: > > > > > > > > > Yes. It’s just a different partition on the same nfs server. > > > > > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller > > > > > > > > > > wrote: > > > > > The new and old primary storage is in the same zone, correct? > > > > > Did you also change out the secondary storage? > > > > > > > > > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen > > > > > > > > > > wrote: > > > > > > > > > > Yes. On Kvm. I’ve been trying to destroy them from the interface and > > > > > it > > > > > just keep churning. I did a destroy with virsh, but no status changed > > > > > in > > > > > the interface. Also, the newly created ones don’t seem to bring up > > > > > their > > > > > agent and never fully start. > > > > > > > > > > Thanks > > >
Re: Stuck in Preparing for maintenance on primary storage
The node cloudstack is claiming the system vm’s is starting on shows no signs of any vm’s running. virsh list is black. Thanks -jeremy > On Tuesday, Feb 21, 2023 at 8:23 PM, Me (mailto:jer...@skidrow.la)> wrote: > Also, just to note, I’m not sure how much made it in to the logs. The system > vm’s are stuck in starting state and trying to kill through the interface > doesn’t seem to do anything. > > -jeremy > > > > > > On Tuesday, Feb 21, 2023 at 8:20 PM, Me > (mailto:jer...@skidrow.la)> wrote: > > Is there something else I can use to submit logs? Too much for pastebin. > > > > Thanks > > -jeremy > > > > > > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller > > (mailto:siwelle...@gmail.com)> wrote: > > > Can you pull some management server logs and also put the CloudStack KVM > > > agent into debug mode before destroying the ssvm and share the logs? > > > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > > > > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen > > > wrote: > > > > > > > Yes. It’s just a different partition on the same nfs server. > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller > > > > wrote: > > > > The new and old primary storage is in the same zone, correct? > > > > Did you also change out the secondary storage? > > > > > > > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen > > > > wrote: > > > > > > > > Yes. On Kvm. I’ve been trying to destroy them from the interface and it > > > > just keep churning. I did a destroy with virsh, but no status changed in > > > > the interface. Also, the newly created ones don’t seem to bring up their > > > > agent and never fully start. > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller > > > > wrote: > > > > Just destroy the old system VMs and they will be recreated on available > > > > storage. > > > > > > > > Are you on KVM? > > > > > > > > > > > > > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen > > > > wrote: > > > > > > > > How do I completely recreate the system vm? > > > > > > > > I was able to get the old storage in to full maintenance and deleted it, > > > > so maybe the system vm are still using the old storage? Is there a way > > > > to > > > > tell the system vm’s to use the new storage? Db change? > > > > > > > > Thanks! > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller > > > > wrote: > > > > Hey Jeremy, > > > > > > > > Is there anything in the management logs that indicate why it's not > > > > completing the maintenance action? > > > > Usually, this state is triggered by some stuck VMs that haven't migrated > > > > off of the primary storage. > > > > > > > > You mentioned the system VMs. Are they still on the old storage? Could > > > > this > > > > be due to some storage tags? > > > > > > > > -Si > > > > > > > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen > > > > > > > > wrote: > > > > > > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > > > and I can’t remove the old primary storage. > > > > > > > > -jeremy > > > > > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me wrote: > > > > I tried to put one of my primary storage definitions in to maintenance > > > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how > > > > to > > > > remedy this situation: > > > > > > > > Cancel maintenance mode > > > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > > > with id 1 is not ready to complete migration, as the status > > > > is:PrepareForMaintenance > > > > > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > > > to migrate to a new volume. All volumes were moved to the new storage. I > > > > was simply trying to delete the old storage definition. > > > > > > > > Thanks > > > > -jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
Also, just to note, I’m not sure how much made it in to the logs. The system vm’s are stuck in starting state and trying to kill through the interface doesn’t seem to do anything. -jeremy > On Tuesday, Feb 21, 2023 at 8:20 PM, Me (mailto:jer...@skidrow.la)> wrote: > Is there something else I can use to submit logs? Too much for pastebin. > > Thanks > -jeremy > > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller > (mailto:siwelle...@gmail.com)> wrote: > > Can you pull some management server logs and also put the CloudStack KVM > > agent into debug mode before destroying the ssvm and share the logs? > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen > > wrote: > > > > > Yes. It’s just a different partition on the same nfs server. > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller > > > wrote: > > > The new and old primary storage is in the same zone, correct? > > > Did you also change out the secondary storage? > > > > > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen > > > wrote: > > > > > > Yes. On Kvm. I’ve been trying to destroy them from the interface and it > > > just keep churning. I did a destroy with virsh, but no status changed in > > > the interface. Also, the newly created ones don’t seem to bring up their > > > agent and never fully start. > > > > > > Thanks > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller > > > wrote: > > > Just destroy the old system VMs and they will be recreated on available > > > storage. > > > > > > Are you on KVM? > > > > > > > > > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen > > > wrote: > > > > > > How do I completely recreate the system vm? > > > > > > I was able to get the old storage in to full maintenance and deleted it, > > > so maybe the system vm are still using the old storage? Is there a way to > > > tell the system vm’s to use the new storage? Db change? > > > > > > Thanks! > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller > > > wrote: > > > Hey Jeremy, > > > > > > Is there anything in the management logs that indicate why it's not > > > completing the maintenance action? > > > Usually, this state is triggered by some stuck VMs that haven't migrated > > > off of the primary storage. > > > > > > You mentioned the system VMs. Are they still on the old storage? Could > > > this > > > be due to some storage tags? > > > > > > -Si > > > > > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen > > > wrote: > > > > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > > and I can’t remove the old primary storage. > > > > > > -jeremy > > > > > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me wrote: > > > I tried to put one of my primary storage definitions in to maintenance > > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > > > remedy this situation: > > > > > > Cancel maintenance mode > > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > > with id 1 is not ready to complete migration, as the status > > > is:PrepareForMaintenance > > > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > > to migrate to a new volume. All volumes were moved to the new storage. I > > > was simply trying to delete the old storage definition. > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
Is there something else I can use to submit logs? Too much for pastebin. Thanks -jeremy > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller (mailto:siwelle...@gmail.com)> wrote: > Can you pull some management server logs and also put the CloudStack KVM > agent into debug mode before destroying the ssvm and share the logs? > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen > wrote: > > > Yes. It’s just a different partition on the same nfs server. > > > > > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller > > wrote: > > The new and old primary storage is in the same zone, correct? > > Did you also change out the secondary storage? > > > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen > > wrote: > > > > Yes. On Kvm. I’ve been trying to destroy them from the interface and it > > just keep churning. I did a destroy with virsh, but no status changed in > > the interface. Also, the newly created ones don’t seem to bring up their > > agent and never fully start. > > > > Thanks > > > > > > > > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller > > wrote: > > Just destroy the old system VMs and they will be recreated on available > > storage. > > > > Are you on KVM? > > > > > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen > > wrote: > > > > How do I completely recreate the system vm? > > > > I was able to get the old storage in to full maintenance and deleted it, > > so maybe the system vm are still using the old storage? Is there a way to > > tell the system vm’s to use the new storage? Db change? > > > > Thanks! > > > > > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller > > wrote: > > Hey Jeremy, > > > > Is there anything in the management logs that indicate why it's not > > completing the maintenance action? > > Usually, this state is triggered by some stuck VMs that haven't migrated > > off of the primary storage. > > > > You mentioned the system VMs. Are they still on the old storage? Could > > this > > be due to some storage tags? > > > > -Si > > > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen > > wrote: > > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > and I can’t remove the old primary storage. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me wrote: > > I tried to put one of my primary storage definitions in to maintenance > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > > remedy this situation: > > > > Cancel maintenance mode > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > with id 1 is not ready to complete migration, as the status > > is:PrepareForMaintenance > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > to migrate to a new volume. All volumes were moved to the new storage. I > > was simply trying to delete the old storage definition. > > > > Thanks > > -jeremy > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
Yes. It’s just a different partition on the same nfs server. > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller (mailto:siwelle...@gmail.com)> wrote: > The new and old primary storage is in the same zone, correct? > Did you also change out the secondary storage? > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen > wrote: > > > Yes. On Kvm. I’ve been trying to destroy them from the interface and it > > just keep churning. I did a destroy with virsh, but no status changed in > > the interface. Also, the newly created ones don’t seem to bring up their > > agent and never fully start. > > > > Thanks > > > > > > > > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller > > wrote: > > Just destroy the old system VMs and they will be recreated on available > > storage. > > > > Are you on KVM? > > > > > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen > > wrote: > > > > How do I completely recreate the system vm? > > > > I was able to get the old storage in to full maintenance and deleted it, > > so maybe the system vm are still using the old storage? Is there a way to > > tell the system vm’s to use the new storage? Db change? > > > > Thanks! > > > > > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller > > wrote: > > Hey Jeremy, > > > > Is there anything in the management logs that indicate why it's not > > completing the maintenance action? > > Usually, this state is triggered by some stuck VMs that haven't migrated > > off of the primary storage. > > > > You mentioned the system VMs. Are they still on the old storage? Could > > this > > be due to some storage tags? > > > > -Si > > > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen > > wrote: > > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > and I can’t remove the old primary storage. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me wrote: > > I tried to put one of my primary storage definitions in to maintenance > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > > remedy this situation: > > > > Cancel maintenance mode > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > with id 1 is not ready to complete migration, as the status > > is:PrepareForMaintenance > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > to migrate to a new volume. All volumes were moved to the new storage. I > > was simply trying to delete the old storage definition. > > > > Thanks > > -jeremy > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
Yes. On Kvm. I’ve been trying to destroy them from the interface and it just keep churning. I did a destroy with virsh, but no status changed in the interface. Also, the newly created ones don’t seem to bring up their agent and never fully start. Thanks > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller (mailto:siwelle...@gmail.com)> wrote: > Just destroy the old system VMs and they will be recreated on available > storage. > > Are you on KVM? > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen > wrote: > > > How do I completely recreate the system vm? > > > > I was able to get the old storage in to full maintenance and deleted it, > > so maybe the system vm are still using the old storage? Is there a way to > > tell the system vm’s to use the new storage? Db change? > > > > Thanks! > > > > > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller > > wrote: > > Hey Jeremy, > > > > Is there anything in the management logs that indicate why it's not > > completing the maintenance action? > > Usually, this state is triggered by some stuck VMs that haven't migrated > > off of the primary storage. > > > > You mentioned the system VMs. Are they still on the old storage? Could > > this > > be due to some storage tags? > > > > -Si > > > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen > > wrote: > > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > and I can’t remove the old primary storage. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me wrote: > > I tried to put one of my primary storage definitions in to maintenance > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > > remedy this situation: > > > > Cancel maintenance mode > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > with id 1 is not ready to complete migration, as the status > > is:PrepareForMaintenance > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > to migrate to a new volume. All volumes were moved to the new storage. I > > was simply trying to delete the old storage definition. > > > > Thanks > > -jeremy > > > > > > > > > > signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
How do I completely recreate the system vm? I was able to get the old storage in to full maintenance and deleted it, so maybe the system vm are still using the old storage? Is there a way to tell the system vm’s to use the new storage? Db change? Thanks! > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller (mailto:siwelle...@gmail.com)> wrote: > Hey Jeremy, > > Is there anything in the management logs that indicate why it's not > completing the maintenance action? > Usually, this state is triggered by some stuck VMs that haven't migrated > off of the primary storage. > > You mentioned the system VMs. Are they still on the old storage? Could this > be due to some storage tags? > > -Si > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen > wrote: > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > and I can’t remove the old primary storage. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me wrote: > > I tried to put one of my primary storage definitions in to maintenance > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > > remedy this situation: > > > > Cancel maintenance mode > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > with id 1 is not ready to complete migration, as the status > > is:PrepareForMaintenance > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > to migrate to a new volume. All volumes were moved to the new storage. I > > was simply trying to delete the old storage definition. > > > > Thanks > > -jeremy > > > > > > > > signature.asc Description: PGP signature
Re: Stuck in Preparing for maintenance on primary storage
Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s and I can’t remove the old primary storage. -jeremy > On Tuesday, Feb 21, 2023 at 2:35 AM, Me (mailto:jer...@skidrow.la)> wrote: > I tried to put one of my primary storage definitions in to maintenance mode. > Now it’s stuck in preparing for maintenance and I’m not sure how to remedy > this situation: > > Cancel maintenance mode > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage with > id 1 is not ready to complete migration, as the status > is:PrepareForMaintenance > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting to > migrate to a new volume. All volumes were moved to the new storage. I was > simply trying to delete the old storage definition. > > Thanks > -jeremy > > > signature.asc Description: PGP signature
Stuck in Preparing for maintenance on primary storage
I tried to put one of my primary storage definitions in to maintenance mode. Now it’s stuck in preparing for maintenance and I’m not sure how to remedy this situation: Cancel maintenance mode (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage with id 1 is not ready to complete migration, as the status is:PrepareForMaintenance Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… 4.17.2.0. Using NFS for primary and secondary storage. I was attempting to migrate to a new volume. All volumes were moved to the new storage. I was simply trying to delete the old storage definition. Thanks -jeremy signature.asc Description: PGP signature
Re: Updating storage addresses
Got it! Thank you. -jeremy > On Tuesday, Feb 14, 2023 at 2:54 AM, Granwille Strauss (mailto:granwi...@namhost.com)> wrote: > > Hi Jeremy > > > The best would be to add both primary and secondary storages, perhaps look at > giving new NFS paths as well, since it seems to sound like the same storage > devices. Once you have added them, you an use the "Complete" migration policy > option for secondary storage to migrate data between each other. And for > primary storage, you can move the volumes of your instances to the new > storage via UI. Keep in mind you may want to power down the instance before > doing so. > > > Once done, delete your old secondary storage and maybe try rebooting ssvm and > consolevm (May need to destroy but reboot first). You can then also delete > old primary storage once you have confirmed all volumes have been moved > accordingly. > > > On 2/14/23 11:00, Jeremy Hansen wrote: > > I have to update the primary and secondary storage address just to a new IP > > for NFS. I can’t seem to find a way to do this through the interface. Do I > > have to do an update to the db? Any notes on how to do that? > > > > Thanks > > -jeremy > > > > > > > -- > Regards / Groete > > Granwille Strauss // Senior Systems Admin > > e: granwi...@namhost.com (mailto:granwi...@namhost.com) > m: +264 81 323 1260 (tel:+264813231260) > w: www.namhost.com (https://www.namhost.com/) > > > > > Namhost Internet Services (Pty) Ltd, > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this message > to anyone without our explicit consent. The integrity and security of this > email cannot be guaranteed over the Internet. Therefore, the sender will not > be held liable for any damage caused by the message. For our full privacy > policy and disclaimers, please go to https://www.namhost.com/privacy-policy > > > > signature.asc Description: PGP signature
Updating storage addresses
I have to update the primary and secondary storage address just to a new IP for NFS. I can’t seem to find a way to do this through the interface. Do I have to do an update to the db? Any notes on how to do that? Thanks -jeremy signature.asc Description: PGP signature
ata1: lost interrupt
I have a guest vm, kvm based, and I’m seeing this ATA reset in dmesg. When this
happens, performance plummits.
Cloudstack-4.17.1
Guest VM is Rocky 9.1.
[ 1332.734551] ata1: lost interrupt (Status 0x58)
[ 1351.076960] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 1351.077041] ata1.00: failed command: WRITE DMA EXT
[ 1351.077053] ata1.00: cmd 35/00:00:98:9e:dd/00:02:00:00:00/e0 tag 0 dma
262144 out
res 40/00:01:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[ 1351.077101] ata1.00: status: { DRDY }
[ 1351.077325] ata1: soft resetting link
[ 1351.235558] ata1: found unknown device (class 0)
[ 1351.237686] ata1.00: configured for MWDMA2
[ 1351.237733] ata1: EH complete
Since this is the only VM I’m experiencing this issue on, I don’t believe this
is hardware related. I’ve migrated the VM to multiple hosts and the vm carries
this issues wherever it goes.
Backend storage is Ceph.
How would I troubleshoot this?
Thanks
-jeremy
signature.asc
Description: PGP signature
Re: Rocky 9 and CS 4.17.1.0
Thanks. I was trying to get rocky9 as a vm guest, which is now working thanks to the suggestions on the list. I’ll wait to do this for Cloudstack hosts. -jeremy > On Monday, Dec 12, 2022 at 9:36 AM, Rohit Yadav (mailto:rohit.ya...@shapeblue.com)> wrote: > Hi Jeremy, All, > > EL9/Rocky9 isn't supported yet. There is a work-in-progress PR for this > https://github.com/apache/cloudstack/pull/6844 > > > > Regards. > > ________ > > > > From: Jeremy Hansen > Sent: Saturday, December 10, 2022 19:52 > To: Vivek Kumar via users > Subject: Rocky 9 and CS 4.17.1.0 > > I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m having issues > running Rocky 9. Kernel begins to boot and then it looks like it fails on > loading initrd and I get a kernel oops. Just curious if this is a known issue > or if there’s a work around. I tried using the qcow2 image from Rocky as well > and just using the install iso to create a new image. Same result. > > Rocky 8 works fine. > > Anyone running Rocky 9? > > Thanks > -jeremy > > > signature.asc Description: PGP signature
Re: Issues with Rocky 9.1 qcow2 image
This worked of course. Thank you very much! > On Saturday, Dec 10, 2022 at 5:59 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Thank you! > > > > > On Saturday, Dec 10, 2022 at 4:56 PM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi, > > > > You can configure guest cpu model. > > > > Please refer to > > https://docs.cloudstack.apache.org/en/4.17.1.0/installguide/hypervisor/kvm.html#configure-cpu-model-for-kvm-guest-optional > > > > > > > > On Saturday, 10 December 2022, Jeremy Hansen > > wrote: > > > > > Looks like it’s supported by the host hardware on my CS hosts: > > > > > > [root@netman ~]# cexecs cs: "/lib64/ld-linux-x86-64.so.2 --help | grep > > > x86-64-v2" > > > cs > > > - cm01- > > > x86-64-v2 (supported, searched) > > > - cm02- > > > x86-64-v2 (supported, searched) > > > - cn04- > > > x86-64-v2 (supported, searched) > > > - cn05- > > > x86-64-v2 (supported, searched) > > > - cn06- > > > x86-64-v2 (supported, searched) > > > - cn07----- > > > x86-64-v2 (supported, searched) > > > > > > I am running KVM. Any idea how to incorporate using the ‘—cpu host’ > > > option in a Cloudstack environment? > > > > > > Thanks! > > > > > > > > > > > > On Saturday, Dec 10, 2022 at 3:26 PM, Jeremy Hansen > > > wrote: > > > In an effort to troubleshoot this, I decided to launch a Rocky 8 vm and do > > > a manual upgrade to Rocky 9. While I understand this isn’t recommended, I > > > thought perhaps it would reveal what the issue are. After I started doing > > > package upgrade, I noticed this: > > > > > > Fatal glibc error: CPU does not support x86-64-v2 > > > > > > I suspect this is the root of my issues. Can anyone explain this > > > further? Is there something I can change in Cloudstack to allow this v2 > > > capability? > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > On Friday, Dec 09, 2022 at 2:32 PM, Jeremy Hansen > > > wrote: > > > Sorry for the late response. Got covid. Mild. Anyway. > > > > > > I have plenty of Rocky 8 vm’s running with no issue and I can launch news > > > Rocky 8 vm with no issue. Here’s lspci from one of my running rocky 8.7 > > > instances: > > > > > > [root@cmx01 ~]# lspci > > > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev > > > 02) > > > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] > > > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton > > > II] > > > 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton > > > II] (rev 01) > > > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) > > > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 > > > 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device > > > 00:04.0 Communication controller: Red Hat, Inc. Virtio console > > > 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device > > > 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon > > > 00:07.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > On Tuesday, Dec 06, 2022 at 5:10 PM, Nux wrote: > > > What hardware profile are you using? If you boot a system rescue cd or a > > > rocky 8 one, what is the output of "lspci"? > > > > > > On 6 December 2022 18:41:25 GMT, Jeremy Hansen > > > wrote: > > > > > > Unfortunately I’m getting a kernel panic upon boot. I tried both the > > > production grub option and the rescue and I get the same result. Maybe > > > something with the size of the root disk? I have it just at 8 gig right > > > now? Something with the initrd perhaps? > > > > > > Thanks! > > > -jeremy > > > > > > On Tuesday, Dec 06, 2022 at 12:53 PM, Jeremy Hansen > > (mailto:jer...@skidrow.la)> wrote: > > > Wow. Thank you! I appreciate this. I’ll report back after trying. > > > > > > -jeremy > > > > > > > > > > >
Re: Issues with Rocky 9.1 qcow2 image
Thank you! > On Saturday, Dec 10, 2022 at 4:56 PM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi, > > You can configure guest cpu model. > > Please refer to > https://docs.cloudstack.apache.org/en/4.17.1.0/installguide/hypervisor/kvm.html#configure-cpu-model-for-kvm-guest-optional > > > > On Saturday, 10 December 2022, Jeremy Hansen > wrote: > > > Looks like it’s supported by the host hardware on my CS hosts: > > > > [root@netman ~]# cexecs cs: "/lib64/ld-linux-x86-64.so.2 --help | grep > > x86-64-v2" > > cs > > - cm01- > > x86-64-v2 (supported, searched) > > - cm02- > > x86-64-v2 (supported, searched) > > - cn04- > > x86-64-v2 (supported, searched) > > - cn05- > > x86-64-v2 (supported, searched) > > - cn06- > > x86-64-v2 (supported, searched) > > - cn07- > > x86-64-v2 (supported, searched) > > > > I am running KVM. Any idea how to incorporate using the ‘—cpu host’ > > option in a Cloudstack environment? > > > > Thanks! > > > > > > > > On Saturday, Dec 10, 2022 at 3:26 PM, Jeremy Hansen > > wrote: > > In an effort to troubleshoot this, I decided to launch a Rocky 8 vm and do > > a manual upgrade to Rocky 9. While I understand this isn’t recommended, I > > thought perhaps it would reveal what the issue are. After I started doing > > package upgrade, I noticed this: > > > > Fatal glibc error: CPU does not support x86-64-v2 > > > > I suspect this is the root of my issues. Can anyone explain this > > further? Is there something I can change in Cloudstack to allow this v2 > > capability? > > > > Thanks > > -jeremy > > > > > > > > On Friday, Dec 09, 2022 at 2:32 PM, Jeremy Hansen > > wrote: > > Sorry for the late response. Got covid. Mild. Anyway. > > > > I have plenty of Rocky 8 vm’s running with no issue and I can launch news > > Rocky 8 vm with no issue. Here’s lspci from one of my running rocky 8.7 > > instances: > > > > [root@cmx01 ~]# lspci > > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev > > 02) > > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] > > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton > > II] > > 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton > > II] (rev 01) > > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) > > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 > > 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device > > 00:04.0 Communication controller: Red Hat, Inc. Virtio console > > 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device > > 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon > > 00:07.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer > > > > Thanks > > -jeremy > > > > > > > > On Tuesday, Dec 06, 2022 at 5:10 PM, Nux wrote: > > What hardware profile are you using? If you boot a system rescue cd or a > > rocky 8 one, what is the output of "lspci"? > > > > On 6 December 2022 18:41:25 GMT, Jeremy Hansen > > wrote: > > > > Unfortunately I’m getting a kernel panic upon boot. I tried both the > > production grub option and the rescue and I get the same result. Maybe > > something with the size of the root disk? I have it just at 8 gig right > > now? Something with the initrd perhaps? > > > > Thanks! > > -jeremy > > > > On Tuesday, Dec 06, 2022 at 12:53 PM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote: > > Wow. Thank you! I appreciate this. I’ll report back after trying. > > > > -jeremy > > > > > > > > On Tuesday, Dec 06, 2022 at 11:39 AM, Nux > n...@li.nux.ro)> wrote: > > > > > > God knows, I've built you one with the Cloudstack bits enabled, you can > > grab it from below. It's got the root login enabled, so that's your > > default user. Let me know if it works. > > > > http://dl.openvm.eu/cloudstack/rockylinux/9/rockylinux-9-kvm-rootuser. > > qcow2.bz2 > > > > I'll build more later on with non-priviliged users. > > > > --- > > Nux > > www.nux.ro [1] > > > > On 2022-12-06 15:26, Jeremy Hansen wrote: > > > > Unfortunately I think it gets stuck before ever reaching grub. I tr
Re: Issues with Rocky 9.1 qcow2 image
Looks like it’s supported by the host hardware on my CS hosts: [root@netman ~]# cexecs cs: "/lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v2" cs - cm01- x86-64-v2 (supported, searched) - cm02- x86-64-v2 (supported, searched) - cn04- x86-64-v2 (supported, searched) - cn05- x86-64-v2 (supported, searched) - cn06- x86-64-v2 (supported, searched) - cn07- x86-64-v2 (supported, searched) I am running KVM. Any idea how to incorporate using the ‘—cpu host’ option in a Cloudstack environment? Thanks! > On Saturday, Dec 10, 2022 at 3:26 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > In an effort to troubleshoot this, I decided to launch a Rocky 8 vm and do a > manual upgrade to Rocky 9. While I understand this isn’t recommended, I > thought perhaps it would reveal what the issue are. After I started doing > package upgrade, I noticed this: > > Fatal glibc error: CPU does not support x86-64-v2 > > I suspect this is the root of my issues. Can anyone explain this further? Is > there something I can change in Cloudstack to allow this v2 capability? > > Thanks > -jeremy > > > > > > On Friday, Dec 09, 2022 at 2:32 PM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote: > > Sorry for the late response. Got covid. Mild. Anyway. > > > > I have plenty of Rocky 8 vm’s running with no issue and I can launch news > > Rocky 8 vm with no issue. Here’s lspci from one of my running rocky 8.7 > > instances: > > > > [root@cmx01 ~]# lspci > > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) > > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] > > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton > > II] > > 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton > > II] (rev 01) > > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) > > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 > > 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device > > 00:04.0 Communication controller: Red Hat, Inc. Virtio console > > 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device > > 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon > > 00:07.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer > > > > > > Thanks > > -jeremy > > > > > > > > > On Tuesday, Dec 06, 2022 at 5:10 PM, Nux > > (mailto:n...@li.nux.ro)> wrote: > > > What hardware profile are you using? If you boot a system rescue cd or a > > > rocky 8 one, what is the output of "lspci"? > > > > > > On 6 December 2022 18:41:25 GMT, Jeremy Hansen > > > wrote: > > > > Unfortunately I’m getting a kernel panic upon boot. I tried both the > > > > production grub option and the rescue and I get the same result. Maybe > > > > something with the size of the root disk? I have it just at 8 gig right > > > > now? Something with the initrd perhaps? > > > > > > > > Thanks! > > > > -jeremy > > > > > > > > > On Tuesday, Dec 06, 2022 at 12:53 PM, Jeremy Hansen > > > > > mailto:jer...@skidrow.la)> wrote: > > > > > Wow. Thank you! I appreciate this. I’ll report back after trying. > > > > > > > > > > -jeremy > > > > > > > > > > > > > > > > > > > > > On Tuesday, Dec 06, 2022 at 11:39 AM, Nux > > > > > (mailto:n...@li.nux.ro)> wrote: > > > > > > > > > > > > > > > > > > God knows, I've built you one with the Cloudstack bits enabled, you > > > > > > can > > > > > > grab it from below. It's got the root login enabled, so that's your > > > > > > default user. Let me know if it works. > > > > > > > > > > > > http://dl.openvm.eu/cloudstack/rockylinux/9/rockylinux-9-kvm-rootuser.qcow2.bz2 > > > > > > > > > > > > I'll build more later on with non-priviliged users. > > > > > > > > > > > > --- > > > > > > Nux > > > > > > www.nux.ro [1] > > > > > > > > > > > > On 2022-12-06 15:26, Jeremy Hansen wrote: > > > > > > > > > > > > > Unfortunately I think it gets stuck before ever reaching grub. I >
Re: Issues with Rocky 9.1 qcow2 image
In an effort to troubleshoot this, I decided to launch a Rocky 8 vm and do a manual upgrade to Rocky 9. While I understand this isn’t recommended, I thought perhaps it would reveal what the issue are. After I started doing package upgrade, I noticed this: Fatal glibc error: CPU does not support x86-64-v2 I suspect this is the root of my issues. Can anyone explain this further? Is there something I can change in Cloudstack to allow this v2 capability? Thanks -jeremy > On Friday, Dec 09, 2022 at 2:32 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Sorry for the late response. Got covid. Mild. Anyway. > > I have plenty of Rocky 8 vm’s running with no issue and I can launch news > Rocky 8 vm with no issue. Here’s lspci from one of my running rocky 8.7 > instances: > > [root@cmx01 ~]# lspci > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] > 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton > II] (rev 01) > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 > 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device > 00:04.0 Communication controller: Red Hat, Inc. Virtio console > 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device > 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon > 00:07.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer > > > Thanks > -jeremy > > > > > On Tuesday, Dec 06, 2022 at 5:10 PM, Nux > (mailto:n...@li.nux.ro)> wrote: > > What hardware profile are you using? If you boot a system rescue cd or a > > rocky 8 one, what is the output of "lspci"? > > > > On 6 December 2022 18:41:25 GMT, Jeremy Hansen > > wrote: > > > Unfortunately I’m getting a kernel panic upon boot. I tried both the > > > production grub option and the rescue and I get the same result. Maybe > > > something with the size of the root disk? I have it just at 8 gig right > > > now? Something with the initrd perhaps? > > > > > > Thanks! > > > -jeremy > > > > > > > On Tuesday, Dec 06, 2022 at 12:53 PM, Jeremy Hansen > > > (mailto:jer...@skidrow.la)> wrote: > > > > Wow. Thank you! I appreciate this. I’ll report back after trying. > > > > > > > > -jeremy > > > > > > > > > > > > > > > > > On Tuesday, Dec 06, 2022 at 11:39 AM, Nux > > > > (mailto:n...@li.nux.ro)> wrote: > > > > > > > > > > > > > > > God knows, I've built you one with the Cloudstack bits enabled, you > > > > > can > > > > > grab it from below. It's got the root login enabled, so that's your > > > > > default user. Let me know if it works. > > > > > > > > > > http://dl.openvm.eu/cloudstack/rockylinux/9/rockylinux-9-kvm-rootuser.qcow2.bz2 > > > > > > > > > > I'll build more later on with non-priviliged users. > > > > > > > > > > --- > > > > > Nux > > > > > www.nux.ro [1] > > > > > > > > > > On 2022-12-06 15:26, Jeremy Hansen wrote: > > > > > > > > > > > Unfortunately I think it gets stuck before ever reaching grub. I > > > > > > tried > > > > > > to catch it on the console after a reboot and this is all I see. I > > > > > > never get to the grub menu. > > > > > > > > > > > > Is there a way I can alter the qcow2 image before I create a > > > > > > template? > > > > > > > > > > > > I thought perhaps I could see the IP it obtains via dhcp, but I see > > > > > > nothing coming from that VM which tells me it's not getting to the > > > > > > point of bringing up its interface. > > > > > > > > > > > > Thanks > > > > > > -jeremy > > > > > > > > > > > > On Tuesday, Dec 06, 2022 at 4:59 AM, Nux wrote: > > > > > > > > > > > > You can't alter the kernel parameters from libvirt as the image > > > > > > loads > > > > > > its own kernel, but you can alter the parameters at their grub > > > > > > screen. > > > > > > Try to open the console pro
Rocky 9 and CS 4.17.1.0
I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m having issues running Rocky 9. Kernel begins to boot and then it looks like it fails on loading initrd and I get a kernel oops. Just curious if this is a known issue or if there’s a work around. I tried using the qcow2 image from Rocky as well and just using the install iso to create a new image. Same result. Rocky 8 works fine. Anyone running Rocky 9? Thanks -jeremy signature.asc Description: PGP signature
Re: Issues with Rocky 9.1 qcow2 image
Sorry for the late response. Got covid. Mild. Anyway. I have plenty of Rocky 8 vm’s running with no issue and I can launch news Rocky 8 vm with no issue. Here’s lspci from one of my running rocky 8.7 instances: [root@cmx01 ~]# lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device 00:04.0 Communication controller: Red Hat, Inc. Virtio console 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon 00:07.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer Thanks -jeremy > On Tuesday, Dec 06, 2022 at 5:10 PM, Nux (mailto:n...@li.nux.ro)> wrote: > What hardware profile are you using? If you boot a system rescue cd or a > rocky 8 one, what is the output of "lspci"? > > On 6 December 2022 18:41:25 GMT, Jeremy Hansen > wrote: > > Unfortunately I’m getting a kernel panic upon boot. I tried both the > > production grub option and the rescue and I get the same result. Maybe > > something with the size of the root disk? I have it just at 8 gig right > > now? Something with the initrd perhaps? > > > > Thanks! > > -jeremy > > > > > On Tuesday, Dec 06, 2022 at 12:53 PM, Jeremy Hansen > > (mailto:jer...@skidrow.la)> wrote: > > > Wow. Thank you! I appreciate this. I’ll report back after trying. > > > > > > -jeremy > > > > > > > > > > > > > On Tuesday, Dec 06, 2022 at 11:39 AM, Nux > > > (mailto:n...@li.nux.ro)> wrote: > > > > > > > > > > > > God knows, I've built you one with the Cloudstack bits enabled, you can > > > > grab it from below. It's got the root login enabled, so that's your > > > > default user. Let me know if it works. > > > > > > > > http://dl.openvm.eu/cloudstack/rockylinux/9/rockylinux-9-kvm-rootuser.qcow2.bz2 > > > > > > > > I'll build more later on with non-priviliged users. > > > > > > > > --- > > > > Nux > > > > www.nux.ro [1] > > > > > > > > On 2022-12-06 15:26, Jeremy Hansen wrote: > > > > > > > > > Unfortunately I think it gets stuck before ever reaching grub. I tried > > > > > to catch it on the console after a reboot and this is all I see. I > > > > > never get to the grub menu. > > > > > > > > > > Is there a way I can alter the qcow2 image before I create a template? > > > > > > > > > > I thought perhaps I could see the IP it obtains via dhcp, but I see > > > > > nothing coming from that VM which tells me it's not getting to the > > > > > point of bringing up its interface. > > > > > > > > > > Thanks > > > > > -jeremy > > > > > > > > > > On Tuesday, Dec 06, 2022 at 4:59 AM, Nux wrote: > > > > > > > > > > You can't alter the kernel parameters from libvirt as the image loads > > > > > its own kernel, but you can alter the parameters at their grub screen. > > > > > Try to open the console proxy as soon as possible and hit up or down > > > > > arrow so grub doesn't proceed, then hit "e" key (I think) to edit the > > > > > appropriate vmlinuz entry. > > > > > > > > > > My hunch is that this being a generic image (made mostly with EC2 and > > > > > possibly Openstack in mind) they send all output to a serial console > > > > > instead of vga, so you would need to remove any parameters such as > > > > > console=ttyS0 and then boot it. > > > > > > > > > > --- > > > > > Nux > > > > > www.nux.ro [1] > > > > > > > > > > On 2022-12-06 04:26, Jeremy Hansen wrote: > > > > > > > > > > I'm trying to boot a Rocky cloud image for Rocky 9.1 using their qcow2 > > > > > image. > > > > > > > > > > As soon as I start the VM, I get: > > > > > > > > > > "Probing EDD (edd=off to disable)... ok" > > > > > > > > > > and then nothing. I don't see a DHCP lease being pulled, so I assume > > > > > the VM isn't actually proceeding with boot. > > > > > > > > > > Anyone else see this? This is on Cloudstack 4.17.1.0. I posted a > > > > > similar message on the Rocky mail list. Is there an easy way via virsh > > > > > or cloudstack to alter the kernel command line parameters in an > > > > > existing image before it boots? > > > > > > > > > > Thanks > > > > > -jeremy > > > > > > > > > > Links: > > > > > -- > > > > > [1] http://www.nux.ro > > > > > > > > > > > > Links: > > > > -- > > > > [1] http://www.nux.ro signature.asc Description: PGP signature
Re: Issues with Rocky 9.1 qcow2 image
Strange, I tried launching the Rocky 9.1 ISO to build a new image and it also kernel panics upon boot. Not sure what the issue is. My Rocky 8 images seem fine. Hmmm… -jeremy > On Tuesday, Dec 06, 2022 at 1:41 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Unfortunately I’m getting a kernel panic upon boot. I tried both the > production grub option and the rescue and I get the same result. Maybe > something with the size of the root disk? I have it just at 8 gig right now? > Something with the initrd perhaps? > > Thanks! > -jeremy > > > > > > > On Tuesday, Dec 06, 2022 at 12:53 PM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote: > > Wow. Thank you! I appreciate this. I’ll report back after trying. > > > > -jeremy > > > > > > > > > On Tuesday, Dec 06, 2022 at 11:39 AM, Nux > > (mailto:n...@li.nux.ro)> wrote: > > > > > > > > > God knows, I've built you one with the Cloudstack bits enabled, you can > > > grab it from below. It's got the root login enabled, so that's your > > > default user. Let me know if it works. > > > > > > http://dl.openvm.eu/cloudstack/rockylinux/9/rockylinux-9-kvm-rootuser.qcow2.bz2 > > > > > > I'll build more later on with non-priviliged users. > > > > > > --- > > > Nux > > > www.nux.ro [1] > > > > > > On 2022-12-06 15:26, Jeremy Hansen wrote: > > > > > > > Unfortunately I think it gets stuck before ever reaching grub. I tried > > > > to catch it on the console after a reboot and this is all I see. I > > > > never get to the grub menu. > > > > > > > > Is there a way I can alter the qcow2 image before I create a template? > > > > > > > > I thought perhaps I could see the IP it obtains via dhcp, but I see > > > > nothing coming from that VM which tells me it's not getting to the > > > > point of bringing up its interface. > > > > > > > > Thanks > > > > -jeremy > > > > > > > > On Tuesday, Dec 06, 2022 at 4:59 AM, Nux wrote: > > > > > > > > You can't alter the kernel parameters from libvirt as the image loads > > > > its own kernel, but you can alter the parameters at their grub screen. > > > > Try to open the console proxy as soon as possible and hit up or down > > > > arrow so grub doesn't proceed, then hit "e" key (I think) to edit the > > > > appropriate vmlinuz entry. > > > > > > > > My hunch is that this being a generic image (made mostly with EC2 and > > > > possibly Openstack in mind) they send all output to a serial console > > > > instead of vga, so you would need to remove any parameters such as > > > > console=ttyS0 and then boot it. > > > > > > > > --- > > > > Nux > > > > www.nux.ro [1] > > > > > > > > On 2022-12-06 04:26, Jeremy Hansen wrote: > > > > > > > > I'm trying to boot a Rocky cloud image for Rocky 9.1 using their qcow2 > > > > image. > > > > > > > > As soon as I start the VM, I get: > > > > > > > > "Probing EDD (edd=off to disable)... ok" > > > > > > > > and then nothing. I don't see a DHCP lease being pulled, so I assume > > > > the VM isn't actually proceeding with boot. > > > > > > > > Anyone else see this? This is on Cloudstack 4.17.1.0. I posted a > > > > similar message on the Rocky mail list. Is there an easy way via virsh > > > > or cloudstack to alter the kernel command line parameters in an > > > > existing image before it boots? > > > > > > > > Thanks > > > > -jeremy > > > > > > > > Links: > > > > -- > > > > [1] http://www.nux.ro > > > > > > > > > Links: > > > -- > > > [1] http://www.nux.ro signature.asc Description: PGP signature
Re: Issues with Rocky 9.1 qcow2 image
Wow. Thank you! I appreciate this. I’ll report back after trying. -jeremy > On Tuesday, Dec 06, 2022 at 11:39 AM, Nux (mailto:n...@li.nux.ro)> wrote: > > > God knows, I've built you one with the Cloudstack bits enabled, you can > grab it from below. It's got the root login enabled, so that's your > default user. Let me know if it works. > > http://dl.openvm.eu/cloudstack/rockylinux/9/rockylinux-9-kvm-rootuser.qcow2.bz2 > > I'll build more later on with non-priviliged users. > > --- > Nux > www.nux.ro [1] > > On 2022-12-06 15:26, Jeremy Hansen wrote: > > > Unfortunately I think it gets stuck before ever reaching grub. I tried > > to catch it on the console after a reboot and this is all I see. I > > never get to the grub menu. > > > > Is there a way I can alter the qcow2 image before I create a template? > > > > I thought perhaps I could see the IP it obtains via dhcp, but I see > > nothing coming from that VM which tells me it's not getting to the > > point of bringing up its interface. > > > > Thanks > > -jeremy > > > > On Tuesday, Dec 06, 2022 at 4:59 AM, Nux wrote: > > > > You can't alter the kernel parameters from libvirt as the image loads > > its own kernel, but you can alter the parameters at their grub screen. > > Try to open the console proxy as soon as possible and hit up or down > > arrow so grub doesn't proceed, then hit "e" key (I think) to edit the > > appropriate vmlinuz entry. > > > > My hunch is that this being a generic image (made mostly with EC2 and > > possibly Openstack in mind) they send all output to a serial console > > instead of vga, so you would need to remove any parameters such as > > console=ttyS0 and then boot it. > > > > --- > > Nux > > www.nux.ro [1] > > > > On 2022-12-06 04:26, Jeremy Hansen wrote: > > > > I'm trying to boot a Rocky cloud image for Rocky 9.1 using their qcow2 > > image. > > > > As soon as I start the VM, I get: > > > > "Probing EDD (edd=off to disable)... ok" > > > > and then nothing. I don't see a DHCP lease being pulled, so I assume > > the VM isn't actually proceeding with boot. > > > > Anyone else see this? This is on Cloudstack 4.17.1.0. I posted a > > similar message on the Rocky mail list. Is there an easy way via virsh > > or cloudstack to alter the kernel command line parameters in an > > existing image before it boots? > > > > Thanks > > -jeremy > > > > Links: > > -- > > [1] http://www.nux.ro > > > Links: > -- > [1] http://www.nux.ro signature.asc Description: PGP signature
Re: Issues with Rocky 9.1 qcow2 image
Unfortunately I think it gets stuck before ever reaching grub. I tried to catch it on the console after a reboot and this is all I see. I never get to the grub menu. Is there a way I can alter the qcow2 image before I create a template? I thought perhaps I could see the IP it obtains via dhcp, but I see nothing coming from that VM which tells me it’s not getting to the point of bringing up its interface. Thanks -jeremy > On Tuesday, Dec 06, 2022 at 4:59 AM, Nux (mailto:n...@li.nux.ro)> wrote: > > > You can't alter the kernel parameters from libvirt as the image loads > its own kernel, but you can alter the parameters at their grub screen. > Try to open the console proxy as soon as possible and hit up or down > arrow so grub doesn't proceed, then hit "e" key (I think) to edit the > appropriate vmlinuz entry. > > My hunch is that this being a generic image (made mostly with EC2 and > possibly Openstack in mind) they send all output to a serial console > instead of vga, so you would need to remove any parameters such as > console=ttyS0 and then boot it. > > --- > Nux > www.nux.ro [1] > > On 2022-12-06 04:26, Jeremy Hansen wrote: > > > I'm trying to boot a Rocky cloud image for Rocky 9.1 using their qcow2 > > image. > > > > As soon as I start the VM, I get: > > > > "Probing EDD (edd=off to disable)... ok" > > > > and then nothing. I don't see a DHCP lease being pulled, so I assume > > the VM isn't actually proceeding with boot. > > > > Anyone else see this? This is on Cloudstack 4.17.1.0. I posted a > > similar message on the Rocky mail list. Is there an easy way via virsh > > or cloudstack to alter the kernel command line parameters in an > > existing image before it boots? > > > > Thanks > > -jeremy > > > Links: > -- > [1] http://www.nux.ro signature.asc Description: PGP signature
Issues with Rocky 9.1 qcow2 image
I’m trying to boot a Rocky cloud image for Rocky 9.1 using their qcow2 image. As soon as I start the VM, I get: “Probing EDD (edd=off to disable)… ok” and then nothing. I don’t see a DHCP lease being pulled, so I assume the VM isn’t actually proceeding with boot. Anyone else see this? This is on Cloudstack 4.17.1.0. I posted a similar message on the Rocky mail list. Is there an easy way via virsh or cloudstack to alter the kernel command line parameters in an existing image before it boots? Thanks -jeremy signature.asc Description: PGP signature
Re: Enable available CPU security flaw mitigations
So there’s no way to change that? > On Thursday, Jun 30, 2022 at 11:37 PM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > Yes, threads is set to always 1. > > -Wei > > On Fri, 1 Jul 2022 at 06:46, Jeremy Hansen > wrote: > > > So I was able to figure out how to specify cpu model and capabilities, but > > I’m not seeing a clear way to specify threads, which I would assume would > > be in the Compute Offerings, but I see nothing to that looks obvious > > reguarding threads. > > > > From the specific vm config through virt-manager, I see: > > > > > > kvm64 > > > > > > > > > > > > > > > > > > > > > > I tried: > > > > guest.cpu.mode=custom > > guest.cpu.model=kvm64 > > guest.cpu.topology.threads=4 > > guest.cpu.features=x2apic hypervisor lahf_lm ibpb spec-ctrl ssbd > > > > in agent.properties but the threads config does nothing. I don’t really > > want the threads definition to be host wide, which is why I assumed this > > would be part of the compute offerings. > > > > Thanks! > > -jeremy > > > > > > > > > > On Wednesday, Jun 29, 2022 at 4:48 AM, Wei ZHOU > > wrote: > > Hi Jeremy, > > > > As far as I know, it means the meltdown and spectre which have already > > been > > solved. The issues do not exist with the recent cpu models. > > > > Anyway, you can specify the cpu model (-IBRS) and add cpu features > > (e.g. ibrs) in the agent.properties on kvm hosts. > > Please refer to > > > > http://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#configure-cpu-model-for-kvm-guest-optional > > > > -Wei > > > > > > On Wed, 29 Jun 2022 at 11:27, Jeremy Hansen > > wrote: > > > > Enable available CPU security flaw mitigations. > > > > I noticed this while digging around in virt-manager. How would I enable > > this as a default for all VMs in Cloudstack? > > > > Thanks > > -jeremy > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Enable available CPU security flaw mitigations
So I was able to figure out how to specify cpu model and capabilities, but I’m not seeing a clear way to specify threads, which I would assume would be in the Compute Offerings, but I see nothing to that looks obvious reguarding threads. From the specific vm config through virt-manager, I see: kvm64 I tried: guest.cpu.mode=custom guest.cpu.model=kvm64 guest.cpu.topology.threads=4 guest.cpu.features=x2apic hypervisor lahf_lm ibpb spec-ctrl ssbd in agent.properties but the threads config does nothing. I don’t really want the threads definition to be host wide, which is why I assumed this would be part of the compute offerings. Thanks! -jeremy > On Wednesday, Jun 29, 2022 at 4:48 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > As far as I know, it means the meltdown and spectre which have already been > solved. The issues do not exist with the recent cpu models. > > Anyway, you can specify the cpu model (-IBRS) and add cpu features > (e.g. ibrs) in the agent.properties on kvm hosts. > Please refer to > http://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#configure-cpu-model-for-kvm-guest-optional > > -Wei > > > On Wed, 29 Jun 2022 at 11:27, Jeremy Hansen > wrote: > > > Enable available CPU security flaw mitigations. > > > > I noticed this while digging around in virt-manager. How would I enable > > this as a default for all VMs in Cloudstack? > > > > Thanks > > -jeremy > > > > > > > > > > signature.asc Description: PGP signature
Re: Enable available CPU security flaw mitigations
Thanks. Some of my kvm nodes are using fairly old cpu’s, so it may still apply. -jeremy > On Wednesday, Jun 29, 2022 at 4:48 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > As far as I know, it means the meltdown and spectre which have already been > solved. The issues do not exist with the recent cpu models. > > Anyway, you can specify the cpu model (-IBRS) and add cpu features > (e.g. ibrs) in the agent.properties on kvm hosts. > Please refer to > http://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#configure-cpu-model-for-kvm-guest-optional > > -Wei > > > On Wed, 29 Jun 2022 at 11:27, Jeremy Hansen > wrote: > > > Enable available CPU security flaw mitigations. > > > > I noticed this while digging around in virt-manager. How would I enable > > this as a default for all VMs in Cloudstack? > > > > Thanks > > -jeremy > > > > > > > > > > signature.asc Description: PGP signature
Enable available CPU security flaw mitigations
Enable available CPU security flaw mitigations. I noticed this while digging around in virt-manager. How would I enable this as a default for all VMs in Cloudstack? Thanks -jeremy signature.asc Description: PGP signature
Re: Corruption on SSVM
I assume I would have to destroy the vm and allow it to launch a new one? Will that allocate a new disk image to get rid of whatever corruption I’m seeing? > On Tuesday, Mar 01, 2022 at 6:56 PM, Jeremy Hansen (mailto:jer...@skidrow.la.INVALID)> wrote: > I’m seeing what looks like some file system corruption or block device > corruption on the secondary storage system vm: > > Rebooting seems to fix it temporarily but the issues pop up within 24 hours. > What’s the best way to fix this? > > -jeremy > > > > signature.asc Description: PGP signature
Corruption on SSVM
I’m seeing what looks like some file system corruption or block device corruption on the secondary storage system vm: Rebooting seems to fix it temporarily but the issues pop up within 24 hours. What’s the best way to fix this? -jeremy signature.asc Description: PGP signature
Re: Cloudstack without a router?
Yes. Thank you! I appreciate the help. -jeremy > On Monday, Feb 28, 2022 at 4:15 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > When you create a zone, in the pod setting, you need to input the system Ip > ranges. The private IPs of system vms are picked up from the range. > > Reserved system gateway > Reserved system netmask > Start Reserved system IP > End Reserved system IP > > Please make sure you have input correct values. > > -Wei > > > > On Mon, 28 Feb 2022 at 12:31, Jeremy Hansen > wrote: > > > For example, my secondary storage vm has this assigned: > > > > > > - IP Address > > 192.168.30.62 > > - Private IP Address > > 192.168.30.55 > > > > > > The public IP is predictable because I see this in the config for public > > IP addresses under Network, but the private IP changes each time the vm > > reboots. Where is the 192.168.30.55 coming from? How is it getting > > assigned? Something seems wrong that it requires two separate interfaces > > on the same network. I’m trying to understand what I’m doing wrong. > > > > Thanks > > -jeremy > > > > > > > > > > On Monday, Feb 28, 2022 at 2:56 AM, Jeremy Hansen < > > jer...@skidrow.la.INVALID> wrote: > > I can cut out IPs to use for the system vm’s, but why do they need two > > interfaces? Is there a way to turn off the private interface? > > > > Thanks > > -jeremy > > > > > > > > On Monday, Feb 28, 2022 at 2:50 AM, Wei ZHOU > > wrote: > > Hi Jeremy, > > > > CloudStack must know the static IP of system vms so that they can be > > configured by CloudStack. You should have a better network plan. It is > > easy > > to avoid IP conflicts. > > > > -Wei > > > > On Mon, 28 Feb 2022 at 10:59, Jeremy Hansen > > wrote: > > > > I’m not talking about public as in externally routable IPs. The system vms > > use the terminology of public and private IPs which in my case is just two > > IPs on the same internal subnet so it seems redundant for no real reason. > > In my case public and private is the same network so why have two > > interfaces that are on the same network on each system vm? > > > > I want to control the IPs that get assigned to the system vm’s so I can > > avoid IPs conflicts. I’d like the system vm’s to allocate from the same > > dhcp server the guests vm are pulling from over the L2 network. If it gets > > its ip from dhcp like everything else, I won’t have to worry about IP > > conflicts when the system vm’s seem to just randomly assign IPs that could > > be the ip of another device on the network. I basically just want > > everything to use the dhcp server I’m running external to Cloudstack. > > > > Thanks > > > > > > > > > > > > On Monday, Feb 28, 2022 at 1:31 AM, Nux! wrote: > > > > > > What do you mean by "static IPs"? > > > > The system vms will continue to need the usual networks in Cloudstack. > > > > You will need to look at the "management" and "public" (and "secondary > > storage" if you specified that expressly) networks in Cloudstack, see if > > there are any changes you can do to integrate it in your environment. > > > > Don't forget, none of said networks actually need to use "public" IPs, > > you can operate entirely in the realm of RFC 1918, this may help you > > juggle things around. > > > > --- > > Nux! > > www.nux.ro [1] > > > > On 2022-02-28 07:25, Jeremy Hansen wrote: > > > > One more question related to this. I see System VMs are still using > > static IPs. I'm not sure where they're pulled from since I've removed > > the shared network completely and I'm only using L2 now. > > > > Also, the System VMs have a Public and Private IP, but in my case, > > everything is on a flat network and these interfaces are just getting > > two IPs for the same network. Can I disable one of these interfaces to > > simplify things without breaking things? > > > > Thanks > > -jeremy > > > > On Saturday, Feb 26, 2022 at 3:20 AM, Jeremy Hansen > > wrote: > > > > Figured it out. Thanks again. The L2 network is exactly what I > > needed. > > > > -jeremy > > > > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen > > wrote: > > > > Thank you. I'm working out the L2 config now but it appears to be > > working. My next question, is it possible to tran
Re: Cloudstack without a router?
Thanks. After some digging around, I figured it out and it’s basically exactly what you said. I found what I was looking for under the Traffic Types in the Physical Network config. This is where the ranges are defined for Management/Guest/Public/Storage. I had some really strange ranges defined here mostly due to not knowing what I was doing when I set this up some time ago. It makes a lot more sense now. I limited the ranges available to the system vms and now I can predict the IP being used by the system vms. So I guess it’s not so much the second interface as it was that I had no idea where the private IP was being pulled from. It’s obvious now it’s from the Management range in the Physical Network config. -jeremy > On Monday, Feb 28, 2022 at 3:51 AM, vas...@gmx.de (mailto:vas...@gmx.de)> wrote: > Hi Jeremy, > > Public : Accessible / reachable from outside of the CS environment; are not > controlled and managed by CS. Has nothing to do with the usal terminology > regarding "IP-Adresses". > Private: Internal networks within the cloudstack environment - management / > storage / guesttraffic (if not using "L2 Network" ServiceOfferings). > > I guess you are trying to set up something like what is called > "small-scale" deployment. > which time of zones did you deploy / use for testing? > > The IP adress for system VMs is configured at the zonecreation. you are > providing there a IP address range for systemVMs. This is stored in the CS > - Database. If the machine reboots the system VM will get the ip out the > pool of "assigned" ip adresses. > You should find theses in the networksettings of the zone you have > deployend. There you can configure the range of availeable ips for > systemvms. > > Regards > Chris > > Am Mo., 28. Feb. 2022 um 10:59 Uhr schrieb Jeremy Hansen > : > > > I’m not talking about public as in externally routable IPs. The system vms > > use the terminology of public and private IPs which in my case is just two > > IPs on the same internal subnet so it seems redundant for no real reason. > > In my case public and private is the same network so why have two > > interfaces that are on the same network on each system vm? > > > > I want to control the IPs that get assigned to the system vm’s so I can > > avoid IPs conflicts. I’d like the system vm’s to allocate from the same > > dhcp server the guests vm are pulling from over the L2 network. If it gets > > its ip from dhcp like everything else, I won’t have to worry about IP > > conflicts when the system vm’s seem to just randomly assign IPs that could > > be the ip of another device on the network. I basically just want > > everything to use the dhcp server I’m running external to Cloudstack. > > > > Thanks > > > > > > > > > > > > On Monday, Feb 28, 2022 at 1:31 AM, Nux! wrote: > > > > > > What do you mean by "static IPs"? > > > > The system vms will continue to need the usual networks in Cloudstack. > > > > You will need to look at the "management" and "public" (and "secondary > > storage" if you specified that expressly) networks in Cloudstack, see if > > there are any changes you can do to integrate it in your environment. > > > > Don't forget, none of said networks actually need to use "public" IPs, > > you can operate entirely in the realm of RFC 1918, this may help you > > juggle things around. > > > > --- > > Nux! > > www.nux.ro [1] > > > > On 2022-02-28 07:25, Jeremy Hansen wrote: > > > > One more question related to this. I see System VMs are still using > > static IPs. I'm not sure where they're pulled from since I've removed > > the shared network completely and I'm only using L2 now. > > > > Also, the System VMs have a Public and Private IP, but in my case, > > everything is on a flat network and these interfaces are just getting > > two IPs for the same network. Can I disable one of these interfaces to > > simplify things without breaking things? > > > > Thanks > > -jeremy > > > > On Saturday, Feb 26, 2022 at 3:20 AM, Jeremy Hansen > > wrote: > > > > Figured it out. Thanks again. The L2 network is exactly what I > > needed. > > > > -jeremy > > > > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen > > wrote: > > > > Thank you. I'm working out the L2 config now but it appears to be > > working. My next question, is it possible to transition existing VMs > > to a new guest network? I didn't see anything obvious. Cloudmonkey? > > &
Re: Cloudstack without a router?
For example, my secondary storage vm has this assigned: IP Address 192.168.30.62 Private IP Address 192.168.30.55 The public IP is predictable because I see this in the config for public IP addresses under Network, but the private IP changes each time the vm reboots. Where is the 192.168.30.55 coming from? How is it getting assigned? Something seems wrong that it requires two separate interfaces on the same network. I’m trying to understand what I’m doing wrong. Thanks -jeremy > On Monday, Feb 28, 2022 at 2:56 AM, Jeremy Hansen (mailto:jer...@skidrow.la.INVALID)> wrote: > I can cut out IPs to use for the system vm’s, but why do they need two > interfaces? Is there a way to turn off the private interface? > > Thanks > -jeremy > > > > > On Monday, Feb 28, 2022 at 2:50 AM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi Jeremy, > > > > CloudStack must know the static IP of system vms so that they can be > > configured by CloudStack. You should have a better network plan. It is easy > > to avoid IP conflicts. > > > > -Wei > > > > On Mon, 28 Feb 2022 at 10:59, Jeremy Hansen > > wrote: > > > > > I’m not talking about public as in externally routable IPs. The system vms > > > use the terminology of public and private IPs which in my case is just two > > > IPs on the same internal subnet so it seems redundant for no real reason. > > > In my case public and private is the same network so why have two > > > interfaces that are on the same network on each system vm? > > > > > > I want to control the IPs that get assigned to the system vm’s so I can > > > avoid IPs conflicts. I’d like the system vm’s to allocate from the same > > > dhcp server the guests vm are pulling from over the L2 network. If it gets > > > its ip from dhcp like everything else, I won’t have to worry about IP > > > conflicts when the system vm’s seem to just randomly assign IPs that could > > > be the ip of another device on the network. I basically just want > > > everything to use the dhcp server I’m running external to Cloudstack. > > > > > > Thanks > > > > > > > > > > > > > > > > > > On Monday, Feb 28, 2022 at 1:31 AM, Nux! wrote: > > > > > > > > > What do you mean by "static IPs"? > > > > > > The system vms will continue to need the usual networks in Cloudstack. > > > > > > You will need to look at the "management" and "public" (and "secondary > > > storage" if you specified that expressly) networks in Cloudstack, see if > > > there are any changes you can do to integrate it in your environment. > > > > > > Don't forget, none of said networks actually need to use "public" IPs, > > > you can operate entirely in the realm of RFC 1918, this may help you > > > juggle things around. > > > > > > --- > > > Nux! > > > www.nux.ro [1] > > > > > > On 2022-02-28 07:25, Jeremy Hansen wrote: > > > > > > One more question related to this. I see System VMs are still using > > > static IPs. I'm not sure where they're pulled from since I've removed > > > the shared network completely and I'm only using L2 now. > > > > > > Also, the System VMs have a Public and Private IP, but in my case, > > > everything is on a flat network and these interfaces are just getting > > > two IPs for the same network. Can I disable one of these interfaces to > > > simplify things without breaking things? > > > > > > Thanks > > > -jeremy > > > > > > On Saturday, Feb 26, 2022 at 3:20 AM, Jeremy Hansen > > > wrote: > > > > > > Figured it out. Thanks again. The L2 network is exactly what I > > > needed. > > > > > > -jeremy > > > > > > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen > > > wrote: > > > > > > Thank you. I'm working out the L2 config now but it appears to be > > > working. My next question, is it possible to transition existing VMs > > > to a new guest network? I didn't see anything obvious. Cloudmonkey? > > > > > > -jeremy > > > > > > On Saturday, Feb 26, 2022 at 1:07 AM, Wei ZHOU > > > wrote: > > > HI Jeremy, > > > > > > You can use L2 network. > > > > > > It is not system VMS stealing ip, but might because you set wrong ip > > > range > > > when you added the pod. > > > > > > Wei > > > > > > On Saturday, 26 February 2022, Jeremy Hansen > > > > > > wrote: > > > > > > Is there a way to run Cloudstack without a virtual router? I basically > > > want CS to handle the management of vm's but I'd like to use outside > > > network services for dhcp/ip allocation. Separate dhcp server not > > > managed > > > by CS? Is this possible? > > > > > > How can I dictate the IPs used by infrastructure VMs? I'm running in to > > > IP conflicts because system vm's keep stealing IPs that are already > > > being > > > used. > > > > > > Thanks > > > > > > > > > > > > Links: > > > -- > > > [1] http://www.nux.ro > > > > > > signature.asc Description: PGP signature
Re: Cloudstack without a router?
I can cut out IPs to use for the system vm’s, but why do they need two interfaces? Is there a way to turn off the private interface? Thanks -jeremy > On Monday, Feb 28, 2022 at 2:50 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > CloudStack must know the static IP of system vms so that they can be > configured by CloudStack. You should have a better network plan. It is easy > to avoid IP conflicts. > > -Wei > > On Mon, 28 Feb 2022 at 10:59, Jeremy Hansen > wrote: > > > I’m not talking about public as in externally routable IPs. The system vms > > use the terminology of public and private IPs which in my case is just two > > IPs on the same internal subnet so it seems redundant for no real reason. > > In my case public and private is the same network so why have two > > interfaces that are on the same network on each system vm? > > > > I want to control the IPs that get assigned to the system vm’s so I can > > avoid IPs conflicts. I’d like the system vm’s to allocate from the same > > dhcp server the guests vm are pulling from over the L2 network. If it gets > > its ip from dhcp like everything else, I won’t have to worry about IP > > conflicts when the system vm’s seem to just randomly assign IPs that could > > be the ip of another device on the network. I basically just want > > everything to use the dhcp server I’m running external to Cloudstack. > > > > Thanks > > > > > > > > > > > > On Monday, Feb 28, 2022 at 1:31 AM, Nux! wrote: > > > > > > What do you mean by "static IPs"? > > > > The system vms will continue to need the usual networks in Cloudstack. > > > > You will need to look at the "management" and "public" (and "secondary > > storage" if you specified that expressly) networks in Cloudstack, see if > > there are any changes you can do to integrate it in your environment. > > > > Don't forget, none of said networks actually need to use "public" IPs, > > you can operate entirely in the realm of RFC 1918, this may help you > > juggle things around. > > > > --- > > Nux! > > www.nux.ro [1] > > > > On 2022-02-28 07:25, Jeremy Hansen wrote: > > > > One more question related to this. I see System VMs are still using > > static IPs. I'm not sure where they're pulled from since I've removed > > the shared network completely and I'm only using L2 now. > > > > Also, the System VMs have a Public and Private IP, but in my case, > > everything is on a flat network and these interfaces are just getting > > two IPs for the same network. Can I disable one of these interfaces to > > simplify things without breaking things? > > > > Thanks > > -jeremy > > > > On Saturday, Feb 26, 2022 at 3:20 AM, Jeremy Hansen > > wrote: > > > > Figured it out. Thanks again. The L2 network is exactly what I > > needed. > > > > -jeremy > > > > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen > > wrote: > > > > Thank you. I'm working out the L2 config now but it appears to be > > working. My next question, is it possible to transition existing VMs > > to a new guest network? I didn't see anything obvious. Cloudmonkey? > > > > -jeremy > > > > On Saturday, Feb 26, 2022 at 1:07 AM, Wei ZHOU > > wrote: > > HI Jeremy, > > > > You can use L2 network. > > > > It is not system VMS stealing ip, but might because you set wrong ip > > range > > when you added the pod. > > > > Wei > > > > On Saturday, 26 February 2022, Jeremy Hansen > > > > wrote: > > > > Is there a way to run Cloudstack without a virtual router? I basically > > want CS to handle the management of vm's but I'd like to use outside > > network services for dhcp/ip allocation. Separate dhcp server not > > managed > > by CS? Is this possible? > > > > How can I dictate the IPs used by infrastructure VMs? I'm running in to > > IP conflicts because system vm's keep stealing IPs that are already > > being > > used. > > > > Thanks > > > > > > > > Links: > > -- > > [1] http://www.nux.ro > > > > signature.asc Description: PGP signature
Re: Cloudstack without a router?
I’m not talking about public as in externally routable IPs. The system vms use the terminology of public and private IPs which in my case is just two IPs on the same internal subnet so it seems redundant for no real reason. In my case public and private is the same network so why have two interfaces that are on the same network on each system vm? I want to control the IPs that get assigned to the system vm’s so I can avoid IPs conflicts. I’d like the system vm’s to allocate from the same dhcp server the guests vm are pulling from over the L2 network. If it gets its ip from dhcp like everything else, I won’t have to worry about IP conflicts when the system vm’s seem to just randomly assign IPs that could be the ip of another device on the network. I basically just want everything to use the dhcp server I’m running external to Cloudstack. Thanks > On Monday, Feb 28, 2022 at 1:31 AM, Nux! (mailto:n...@li.nux.ro)> wrote: > > > What do you mean by "static IPs"? > > The system vms will continue to need the usual networks in Cloudstack. > > You will need to look at the "management" and "public" (and "secondary > storage" if you specified that expressly) networks in Cloudstack, see if > there are any changes you can do to integrate it in your environment. > > Don't forget, none of said networks actually need to use "public" IPs, > you can operate entirely in the realm of RFC 1918, this may help you > juggle things around. > > --- > Nux! > www.nux.ro [1] > > On 2022-02-28 07:25, Jeremy Hansen wrote: > > > One more question related to this. I see System VMs are still using > > static IPs. I'm not sure where they're pulled from since I've removed > > the shared network completely and I'm only using L2 now. > > > > Also, the System VMs have a Public and Private IP, but in my case, > > everything is on a flat network and these interfaces are just getting > > two IPs for the same network. Can I disable one of these interfaces to > > simplify things without breaking things? > > > > Thanks > > -jeremy > > > > On Saturday, Feb 26, 2022 at 3:20 AM, Jeremy Hansen > > wrote: > > > > Figured it out. Thanks again. The L2 network is exactly what I > > needed. > > > > -jeremy > > > > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen > > wrote: > > > > Thank you. I'm working out the L2 config now but it appears to be > > working. My next question, is it possible to transition existing VMs > > to a new guest network? I didn't see anything obvious. Cloudmonkey? > > > > -jeremy > > > > On Saturday, Feb 26, 2022 at 1:07 AM, Wei ZHOU > > wrote: > > HI Jeremy, > > > > You can use L2 network. > > > > It is not system VMS stealing ip, but might because you set wrong ip > > range > > when you added the pod. > > > > Wei > > > > On Saturday, 26 February 2022, Jeremy Hansen > > > > wrote: > > > > Is there a way to run Cloudstack without a virtual router? I basically > > want CS to handle the management of vm's but I'd like to use outside > > network services for dhcp/ip allocation. Separate dhcp server not > > managed > > by CS? Is this possible? > > > > How can I dictate the IPs used by infrastructure VMs? I'm running in to > > IP conflicts because system vm's keep stealing IPs that are already > > being > > used. > > > > Thanks > > > Links: > -- > [1] http://www.nux.ro signature.asc Description: PGP signature
Re: Cloudstack without a router?
One more question related to this. I see System VMs are still using static IPs. I’m not sure where they’re pulled from since I’ve removed the shared network completely and I’m only using L2 now. Also, the System VMs have a Public and Private IP, but in my case, everything is on a flat network and these interfaces are just getting two IPs for the same network. Can I disable one of these interfaces to simplify things without breaking things? Thanks -jeremy > On Saturday, Feb 26, 2022 at 3:20 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Figured it out. Thanks again. The L2 network is exactly what I needed. > > -jeremy > > > > > > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote: > > Thank you. I’m working out the L2 config now but it appears to be working. > > My next question, is it possible to transition existing VMs to a new guest > > network? I didn’t see anything obvious. Cloudmonkey? > > > > -jeremy > > > > > > > > > > > On Saturday, Feb 26, 2022 at 1:07 AM, Wei ZHOU > > (mailto:ustcweiz...@gmail.com)> wrote: > > > HI Jeremy, > > > > > > You can use L2 network. > > > > > > It is not system VMS stealing ip, but might because you set wrong ip range > > > when you added the pod. > > > > > > Wei > > > > > > On Saturday, 26 February 2022, Jeremy Hansen > > > wrote: > > > > > > > Is there a way to run Cloudstack without a virtual router? I basically > > > > want CS to handle the management of vm’s but I’d like to use outside > > > > network services for dhcp/ip allocation. Separate dhcp server not > > > > managed > > > > by CS? Is this possible? > > > > > > > > How can I dictate the IPs used by infrastructure VMs? I’m running in to > > > > IP conflicts because system vm’s keep stealing IPs that are already > > > > being > > > > used. > > > > > > > > Thanks > > > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Cloudstack without a router?
Figured it out. Thanks again. The L2 network is exactly what I needed. -jeremy > On Saturday, Feb 26, 2022 at 2:38 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Thank you. I’m working out the L2 config now but it appears to be working. My > next question, is it possible to transition existing VMs to a new guest > network? I didn’t see anything obvious. Cloudmonkey? > > -jeremy > > > > > > On Saturday, Feb 26, 2022 at 1:07 AM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > HI Jeremy, > > > > You can use L2 network. > > > > It is not system VMS stealing ip, but might because you set wrong ip range > > when you added the pod. > > > > Wei > > > > On Saturday, 26 February 2022, Jeremy Hansen > > wrote: > > > > > Is there a way to run Cloudstack without a virtual router? I basically > > > want CS to handle the management of vm’s but I’d like to use outside > > > network services for dhcp/ip allocation. Separate dhcp server not managed > > > by CS? Is this possible? > > > > > > How can I dictate the IPs used by infrastructure VMs? I’m running in to > > > IP conflicts because system vm’s keep stealing IPs that are already being > > > used. > > > > > > Thanks > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Cloudstack without a router?
Thank you. I’m working out the L2 config now but it appears to be working. My next question, is it possible to transition existing VMs to a new guest network? I didn’t see anything obvious. Cloudmonkey? -jeremy > On Saturday, Feb 26, 2022 at 1:07 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > HI Jeremy, > > You can use L2 network. > > It is not system VMS stealing ip, but might because you set wrong ip range > when you added the pod. > > Wei > > On Saturday, 26 February 2022, Jeremy Hansen > wrote: > > > Is there a way to run Cloudstack without a virtual router? I basically > > want CS to handle the management of vm’s but I’d like to use outside > > network services for dhcp/ip allocation. Separate dhcp server not managed > > by CS? Is this possible? > > > > How can I dictate the IPs used by infrastructure VMs? I’m running in to > > IP conflicts because system vm’s keep stealing IPs that are already being > > used. > > > > Thanks > > > > > > > > signature.asc Description: PGP signature
Cloudstack without a router?
Is there a way to run Cloudstack without a virtual router? I basically want CS to handle the management of vm’s but I’d like to use outside network services for dhcp/ip allocation. Separate dhcp server not managed by CS? Is this possible? How can I dictate the IPs used by infrastructure VMs? I’m running in to IP conflicts because system vm’s keep stealing IPs that are already being used. Thanks signature.asc Description: PGP signature
Re: How would I recover from this situation?
I was able to manually fix this in the db: update volumes set state = 'Ready' where uuid='8987c39d-c182-4549-8e30-f06c9e9bdbba’; After this, the vm booted again. I noticed a similar post that mentioned increasing job.cancel.threshold.minutes beyond 60 minutes, which is how I encountered this issue in the first place. Maybe this will allow me to complete the volume migration from nfs -> ceph. Thanks -jeremy > On Sunday, Dec 19, 2021 at 10:53 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Since the ceph image was stuck in “Creating” state, I just removed the > volume. Immediately after removing the volume, I noticed “Migrating” pop up > in the volumes menu for NFS: > > http://www.skidrowstudios.com/ss.png > > Any clue how I can put this back together? > > Thanks > -jeremy > > > > > > On Sunday, Dec 19, 2021 at 5:07 AM, Jeremy Hansen > (mailto:jer...@skidrow.la)> wrote: > > I was attempting to migrate a root filesystem from NFS to Ceph. During the > > process, Cloudstack came back and told me the process took too long and it > > was canceling the job. This put the state of the filesystem in limbo as the > > there no longer the NFS filesystem and the Ceph image is stuck in > > “Creating”. > > > > I was able to export the image from Ceph using "rbd export > > --pool=cloudstack 31c8d8d5-9dde-4512-ab1e-dcce8dbaf6f3 rootfs.img”. > > > > I’m able to mount the /boot filesystem on this image using proper offsets, > > so this gives me the indication that this image is probably healthy, but > > how do I get this back in to Cloudstack and how do I tell the VM to use > > this new image for its root filesystem? This image has an LVM partition and > > it needs to boot in conjunction with the additional storage I provisioned > > for this instance which makes up the lvm volume being used. > > > > I have the image, I just need to get it back in to cloudstack and I need > > the instance config to use this new image. Or may be even simpler, how to I > > re-establish the relationship with the image which now exists in Ceph but > > has no mapping within Cloudstack since the job bailed in the middle? > > > > This is Cloudstack 4.16.0.0 and Ceph Pacific 16.2.4. > > > > Thanks > > -jeremy > > > > > > > > > > > signature.asc Description: PGP signature
Re: How would I recover from this situation?
Since the ceph image was stuck in “Creating” state, I just removed the volume. Immediately after removing the volume, I noticed “Migrating” pop up in the volumes menu for NFS: http://www.skidrowstudios.com/ss.png Any clue how I can put this back together? Thanks -jeremy > On Sunday, Dec 19, 2021 at 5:07 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > I was attempting to migrate a root filesystem from NFS to Ceph. During the > process, Cloudstack came back and told me the process took too long and it > was canceling the job. This put the state of the filesystem in limbo as the > there no longer the NFS filesystem and the Ceph image is stuck in “Creating”. > > I was able to export the image from Ceph using "rbd export --pool=cloudstack > 31c8d8d5-9dde-4512-ab1e-dcce8dbaf6f3 rootfs.img”. > > I’m able to mount the /boot filesystem on this image using proper offsets, so > this gives me the indication that this image is probably healthy, but how do > I get this back in to Cloudstack and how do I tell the VM to use this new > image for its root filesystem? This image has an LVM partition and it needs > to boot in conjunction with the additional storage I provisioned for this > instance which makes up the lvm volume being used. > > I have the image, I just need to get it back in to cloudstack and I need the > instance config to use this new image. Or may be even simpler, how to I > re-establish the relationship with the image which now exists in Ceph but has > no mapping within Cloudstack since the job bailed in the middle? > > This is Cloudstack 4.16.0.0 and Ceph Pacific 16.2.4. > > Thanks > -jeremy > > > > signature.asc Description: PGP signature
How would I recover from this situation?
I was attempting to migrate a root filesystem from NFS to Ceph. During the process, Cloudstack came back and told me the process took too long and it was canceling the job. This put the state of the filesystem in limbo as the there no longer the NFS filesystem and the Ceph image is stuck in “Creating”. I was able to export the image from Ceph using "rbd export --pool=cloudstack 31c8d8d5-9dde-4512-ab1e-dcce8dbaf6f3 rootfs.img”. I’m able to mount the /boot filesystem on this image using proper offsets, so this gives me the indication that this image is probably healthy, but how do I get this back in to Cloudstack and how do I tell the VM to use this new image for its root filesystem? This image has an LVM partition and it needs to boot in conjunction with the additional storage I provisioned for this instance which makes up the lvm volume being used. I have the image, I just need to get it back in to cloudstack and I need the instance config to use this new image. Or may be even simpler, how to I re-establish the relationship with the image which now exists in Ceph but has no mapping within Cloudstack since the job bailed in the middle? This is Cloudstack 4.16.0.0 and Ceph Pacific 16.2.4. Thanks -jeremy signature.asc Description: PGP signature
Re: Issues with libvirt after upgrading to RockyLinux 8.5
Faking the releases file helped me get past this. Thanks > On Thursday, Dec 16, 2021 at 1:32 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Thank you. I thought about trying that. > > If I upgrade everything to Rocky, and I’m on 4.16, should I expect it to > work? Will I have to re-add compute nodes to update the db with Rocky as the > OS? > > I expect a lot of people might want to transition to Rocky since CentOS is > going EOL after this month I believe. At this point, Rocky and CentOS are > mostly the same thing. This complicates things a bit. > > Thanks > -jeremy > > > > > On Thursday, Dec 16, 2021 at 1:09 AM, Andrija Panic > > mailto:andrija.pa...@gmail.com)> wrote: > > Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which > > there are CentOS hosts added > > > > > > Try changing the content of the /etc/*release file - to match the file of > > the CentOS nodes - and then play with restarting agent, and observe if you > > would hit the same error or not - I've seen this issue (RHEL inside CentOS > > cluster, due to bad /etc/rhel-release file content), but I've not tested if > > this "fix" works or not. > > > > Best, > > > > On Thu, 16 Dec 2021 at 08:39, Jeremy Hansen > > wrote: > > > > > But if I convert all the hosts to Rocky and upgrade to 4.16, I should be > > > ok? > > > > > > Thanks > > > > > > > > > > > > On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva < > > > slav...@storpool.com.INVALID> wrote: > > > Sorry, I didn't pay attention to your CS version. After the upgrade, I > > > think you will have the same problem. Because in the DB, there is > > > information about host/hosts on this cluster that is/are with CentOS. > > > > > > Best regards, > > > Slavka > > > > > > On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen > > > wrote: > > > > > > I noticed in the compatibility matrix that Rocky isn’t supported until > > > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still > > > going to complain about the centos/rocky mix? If I convert all my existing > > > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t > > > CentOS and Rocky be considered that same thing… sort of…? > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > > > > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva < > > > slav...@storpool.com.INVALID> wrote: > > > Hi Jeremy, > > > > > > It will help if you have another cluster for Rocky Linux. Hosts need to be > > > of the same OS, it's not possible to mix OSes in the same cluster. > > > > > > Best regards, > > > Slavka > > > > > > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen > > > wrote: > > > > > > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt > > > and qemu and ca-certificates to the same version as the other functional > > > nodes. That didn’t seem to help. This is obviously an ssl issue but I > > > don’t really know what to do about it. > > > > > > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Agent started > > > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Implementation Version is 4.15.0.0 > > > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > > > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Defaulting to using properties file for storage > > > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Defaulting to the constant time backoff algorithm > > > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) > > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > > > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Using default Java settings for IPv6 preference for agent > > > connection > > > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id > > > is 0 > > > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > > file uefi.properties. > > > 2021
Re: Issues with libvirt after upgrading to RockyLinux 8.5
Thank you. I thought about trying that. If I upgrade everything to Rocky, and I’m on 4.16, should I expect it to work? Will I have to re-add compute nodes to update the db with Rocky as the OS? I expect a lot of people might want to transition to Rocky since CentOS is going EOL after this month I believe. At this point, Rocky and CentOS are mostly the same thing. This complicates things a bit. Thanks -jeremy > On Thursday, Dec 16, 2021 at 1:09 AM, Andrija Panic (mailto:andrija.pa...@gmail.com)> wrote: > Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which > there are CentOS hosts added > > > Try changing the content of the /etc/*release file - to match the file of > the CentOS nodes - and then play with restarting agent, and observe if you > would hit the same error or not - I've seen this issue (RHEL inside CentOS > cluster, due to bad /etc/rhel-release file content), but I've not tested if > this "fix" works or not. > > Best, > > On Thu, 16 Dec 2021 at 08:39, Jeremy Hansen > wrote: > > > But if I convert all the hosts to Rocky and upgrade to 4.16, I should be > > ok? > > > > Thanks > > > > > > > > On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva < > > slav...@storpool.com.INVALID> wrote: > > Sorry, I didn't pay attention to your CS version. After the upgrade, I > > think you will have the same problem. Because in the DB, there is > > information about host/hosts on this cluster that is/are with CentOS. > > > > Best regards, > > Slavka > > > > On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen > > wrote: > > > > I noticed in the compatibility matrix that Rocky isn’t supported until > > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still > > going to complain about the centos/rocky mix? If I convert all my existing > > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t > > CentOS and Rocky be considered that same thing… sort of…? > > > > Thanks > > -jeremy > > > > > > > > > > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva < > > slav...@storpool.com.INVALID> wrote: > > Hi Jeremy, > > > > It will help if you have another cluster for Rocky Linux. Hosts need to be > > of the same OS, it's not possible to mix OSes in the same cluster. > > > > Best regards, > > Slavka > > > > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen > > wrote: > > > > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt > > and qemu and ca-certificates to the same version as the other functional > > nodes. That didn’t seem to help. This is obviously an ssl issue but I > > don’t really know what to do about it. > > > > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Agent started > > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Implementation Version is 4.15.0.0 > > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Defaulting to using properties file for storage > > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Defaulting to the constant time backoff algorithm > > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Using default Java settings for IPv6 preference for agent > > connection > > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id > > is 0 > > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > file uefi.properties. > > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null) > > (logid:) No existing libvirtd connection found. Opening a new one > > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > > BridgeVifDriver. > > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) iscsi session clean up is disabled > > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:) > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : > > workers = 5 : host = 192.
Re: Issues with libvirt after upgrading to RockyLinux 8.5
But if I convert all the hosts to Rocky and upgrade to 4.16, I should be ok? Thanks > On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva > mailto:slav...@storpool.com.INVALID)> wrote: > Sorry, I didn't pay attention to your CS version. After the upgrade, I > think you will have the same problem. Because in the DB, there is > information about host/hosts on this cluster that is/are with CentOS. > > Best regards, > Slavka > > On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen > wrote: > > > I noticed in the compatibility matrix that Rocky isn’t supported until > > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still > > going to complain about the centos/rocky mix? If I convert all my existing > > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t > > CentOS and Rocky be considered that same thing… sort of…? > > > > Thanks > > -jeremy > > > > > > > > > > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva < > > slav...@storpool.com.INVALID> wrote: > > Hi Jeremy, > > > > It will help if you have another cluster for Rocky Linux. Hosts need to be > > of the same OS, it's not possible to mix OSes in the same cluster. > > > > Best regards, > > Slavka > > > > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen > > wrote: > > > > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt > > and qemu and ca-certificates to the same version as the other functional > > nodes. That didn’t seem to help. This is obviously an ssl issue but I > > don’t really know what to do about it. > > > > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Agent started > > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Implementation Version is 4.15.0.0 > > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Defaulting to using properties file for storage > > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Defaulting to the constant time backoff algorithm > > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Using default Java settings for IPv6 preference for agent > > connection > > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id > > is 0 > > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > file uefi.properties. > > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null) > > (logid:) No existing libvirtd connection found. Opening a new one > > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > > BridgeVifDriver. > > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) iscsi session clean up is disabled > > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:) > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : > > workers = 5 : host = 192.168.30.59 : port = 8250 > > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:) > > Connecting to 192.168.30.59:8250 > > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf > > file found: /etc/cloudstack/agent/agent.properties > > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:) > > SSL: Handshake done > > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:) > > Connected to 192.168.30.59:8250 > > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt > > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection] > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: > > invalid > > connection pointer in virConnectGetVersion > > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > 18796842-a137-475d-9799-9874240e3c0c, using it. > > 2021-12-15 18:04:16,066
Re: Issues with libvirt after upgrading to RockyLinux 8.5
I noticed in the compatibility matrix that Rocky isn’t supported until 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still going to complain about the centos/rocky mix? If I convert all my existing nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t CentOS and Rocky be considered that same thing… sort of…? Thanks -jeremy > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva > mailto:slav...@storpool.com.INVALID)> wrote: > Hi Jeremy, > > It will help if you have another cluster for Rocky Linux. Hosts need to be > of the same OS, it's not possible to mix OSes in the same cluster. > > Best regards, > Slavka > > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen > wrote: > > > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt > > and qemu and ca-certificates to the same version as the other functional > > nodes. That didn’t seem to help. This is obviously an ssl issue but I > > don’t really know what to do about it. > > > > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Agent started > > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Implementation Version is 4.15.0.0 > > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Defaulting to using properties file for storage > > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Defaulting to the constant time backoff algorithm > > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) > > (logid:) Using default Java settings for IPv6 preference for agent > > connection > > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id > > is 0 > > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > file uefi.properties. > > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null) > > (logid:) No existing libvirtd connection found. Opening a new one > > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > > BridgeVifDriver. > > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) iscsi session clean up is disabled > > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:) > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : > > workers = 5 : host = 192.168.30.59 : port = 8250 > > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:) > > Connecting to 192.168.30.59:8250 > > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf > > file found: /etc/cloudstack/agent/agent.properties > > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:) > > SSL: Handshake done > > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:) > > Connected to 192.168.30.59:8250 > > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt > > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection] > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid > > connection pointer in virConnectGetVersion > > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > 18796842-a137-475d-9799-9874240e3c0c, using it. > > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool > > 18796842-a137-475d-9799-9874240e3c0c from libvirt > > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper] > > (Agent-Handler-1:null) (logid:) Default Builder inited. > > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > (logid:) Proccess agent startup answer, agent id = 0 > > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > (logid:) Set agent id 0 > > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > (logid:) Startup Response Received: agent id = 0 > > 202
Re: Issues with libvirt after upgrading to RockyLinux 8.5
) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source) at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097) at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) The fact that this host has been converted to Rocky Linux is causing an issue? What’s the work around for this? Thanks -jeremy > On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > It doesn’t error out but there is purposely no VMs running on these hosts to > test the upgrade of the underlying distro before having to stop any active > VMs. > > [jeremy@cm02 ~]$ sudo virsh list > Id Name State > > > [jeremy@cm02 ~]$ > > > -jeremy > > > > > > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi, > > > > Do virsh commands e.g. "virsh list" work ? > > > > -Wei > > > > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen > > wrote: > > > > > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5. > > > > > > I’m running Cloudstack 4.15.0.0. > > > > > > Trying to bring up cloudstack agent, I’m seeing some issues communicating > > > with libvirt: > > > > > > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt > > > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection] > > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: > > > invalid > > > connection pointer in virConnectGetVersion > > > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it. > > > > > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output > > > error > > > > > > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64 > > > > > > Functional hosts that have yet to be upgraded are using: > > > > > > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64 > > > > > > > > > My libvirtd.conf looks like this: > > > > > > listen_tcp=0 > > > listen_tls=1 > > > tcp_port="16509" > > > auth_tcp="none" > > > mdns_adv = 0 > > > key_file="/etc/pki/libvirt/private/serverkey.pem" > > > cert_file="/etc/pki/libvirt/servercert.pem" > > > ca_file="/etc/pki/CA/cacert.pem" > > > tls_port="16514" > > > auth_tls=“none" > > > > > > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > > file uefi.properties. > > > 2021-12-12 21:43:42,901 INFO [kvm.resource.L
Re: Issues with libvirt after upgrading to RockyLinux 8.5
It doesn’t error out but there is purposely no VMs running on these hosts to test the upgrade of the underlying distro before having to stop any active VMs. [jeremy@cm02 ~]$ sudo virsh list Id Name State [jeremy@cm02 ~]$ -jeremy > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi, > > Do virsh commands e.g. "virsh list" work ? > > -Wei > > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen > wrote: > > > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5. > > > > I’m running Cloudstack 4.15.0.0. > > > > Trying to bring up cloudstack agent, I’m seeing some issues communicating > > with libvirt: > > > > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt > > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection] > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid > > connection pointer in virConnectGetVersion > > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it. > > > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: > > Input/output error > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: > > Input/output error > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: > > Input/output error > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: > > Input/output error > > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output error > > > > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64 > > > > Functional hosts that have yet to be upgraded are using: > > > > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64 > > > > > > My libvirtd.conf looks like this: > > > > listen_tcp=0 > > listen_tls=1 > > tcp_port="16509" > > auth_tcp="none" > > mdns_adv = 0 > > key_file="/etc/pki/libvirt/private/serverkey.pem" > > cert_file="/etc/pki/libvirt/servercert.pem" > > ca_file="/etc/pki/CA/cacert.pem" > > tls_port="16514" > > auth_tls=“none" > > > > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > file uefi.properties. > > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null) > > (logid:) No existing libvirtd connection found. Opening a new one > > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > > BridgeVifDriver. > > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource] > > (main:null) (logid:) iscsi session clean up is disabled > > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:) > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : > > workers = 5 : host = 192.168.30.59 : port = 8250 > > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:) > > Connecting to 192.168.30.59:8250 > > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf > > file found: /etc/cloudstack/agent/agent.properties > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) > > SSL: Handshake done > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) > > Connected to 192.168.30.59:8250 > > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt > > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection] > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid > > connection pointer in virConnectGetVersion > > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > 18796842-a137-475d-9799-9874240e3c0c, using it. > > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor] > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool > > 18796842-a137-475d-9799-9874240e3c0c from libvirt > > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHel
Issues with libvirt after upgrading to RockyLinux 8.5
Testing on an unused compute node, I tested upgrading to RockyLinux 8.5. I’m running Cloudstack 4.15.0.0. Trying to bring up cloudstack agent, I’m seeing some issues communicating with libvirt: 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool d1b1e853-1c30-473d-badc-6c30318aa5b0, using it. Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: Input/output error Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: Input/output error Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: Input/output error Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: Input/output error Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output error Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64 Functional hosts that have yet to be upgraded are using: libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64 My libvirtd.conf looks like this: listen_tcp=0 listen_tls=1 tcp_port="16509" auth_tcp="none" mdns_adv = 0 key_file="/etc/pki/libvirt/private/serverkey.pem" cert_file="/etc/pki/libvirt/servercert.pem" ca_file="/etc/pki/CA/cacert.pem" tls_port="16514" auth_tls=“none" 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties. 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver. 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.30.59 : port = 8250 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.30.59:8250 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.30.59:8250 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 18796842-a137-475d-9799-9874240e3c0c, using it. 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 18796842-a137-475d-9799-9874240e3c0c from libvirt 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited. 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.15.0.0 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null) (logid:)
Re: Troubleshooting failed volume snapshot?
Well I guess that was the problem. Makes sense. Snapshot worked. Thank you! -jeremy > On Friday, Sep 10, 2021 at 2:10 AM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote: > Thanks. I think downgrading the agents might be easier. I wasn’t quite ready > to upgrade the manager yet. I’ll try that. > > -jeremy > > > > > > On Friday, Sep 10, 2021 at 12:00 AM, Wei ZHOU > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi Jeremy, > > > > Can you upgrade mgt server to 4.15.1.0 (please follow the upgrade > > instructions in cloudstack doc) or downgrade cloudstack agent to 4.15.0.0 , > > and retry ? > > > > -Wei > > > > On Fri, 10 Sept 2021 at 03:34, Jeremy Hansen > > wrote: > > > > > I’m trying to figure out how to debug a snapshot failure. I’m trying to > > > snapshot a root volume and after some time, the process fails but it > > > doesn’t really give any information. > > > > > > Manager host is running cs 4.15.0.0 > > > VM Hosts are running agent version 4.15.1.0 > > > > > > Primary storage is both Ceph and NFS > > > Secondary is just NFS > > > > > > > > > ERROR > > > <http://cm01.cs.la1.clx.corp:8080/client/#/event/2e996766-1e53-4d3e-840c-ef9c2759c777> > > > VOLUME.MIGRATE Completed Error while migrating volume admin > > > <http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > > admin > > > <http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > > ROOT > > > <http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b> > > > 09 > > > Sep 2021 18:01:47 > > > ERROR > > > <http://cm01.cs.la1.clx.corp:8080/client/#/event/39145528-236e-4724-ac59-e3f99f61c624> > > > SNAPSHOT.CREATE Completed Error while taking snapshot admin > > > <http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > > admin > > > <http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > > ROOT > > > <http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b> > > > 09 > > > Sep 2021 17:53:16 > > > ERROR > > > <http://cm01.cs.la1.clx.corp:8080/client/#/event/d830de84-d9da-4756-92e4-1c03a9d39f51> > > > SNAPSHOT.CREATE Completed Error while taking snapshot admin > > > <http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > > admin > > > <http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > > ROOT > > > <http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b> > > > 09 > > > Sep 2021 17:45:06 > > > > > > I’m not really sure where to look to figured out what went wrong. I have > > > VMs servinig from both NFS and Ceph with no issues. I’m able to migrate a > > > volume from NFS to Ceph without any issues but snapshots fail. > > > > > > CentOS 8. > > > > > > Thanks > > > -jeremy > > > > > > > > > > > > > > > > > > > > > signature.asc Description: PGP signature
Re: Troubleshooting failed volume snapshot?
Thanks. I think downgrading the agents might be easier. I wasn’t quite ready to upgrade the manager yet. I’ll try that. -jeremy > On Friday, Sep 10, 2021 at 12:00 AM, Wei ZHOU (mailto:ustcweiz...@gmail.com)> wrote: > Hi Jeremy, > > Can you upgrade mgt server to 4.15.1.0 (please follow the upgrade > instructions in cloudstack doc) or downgrade cloudstack agent to 4.15.0.0 , > and retry ? > > -Wei > > On Fri, 10 Sept 2021 at 03:34, Jeremy Hansen > wrote: > > > I’m trying to figure out how to debug a snapshot failure. I’m trying to > > snapshot a root volume and after some time, the process fails but it > > doesn’t really give any information. > > > > Manager host is running cs 4.15.0.0 > > VM Hosts are running agent version 4.15.1.0 > > > > Primary storage is both Ceph and NFS > > Secondary is just NFS > > > > > > ERROR > > <http://cm01.cs.la1.clx.corp:8080/client/#/event/2e996766-1e53-4d3e-840c-ef9c2759c777> > > VOLUME.MIGRATE Completed Error while migrating volume admin > > <http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > admin > > <http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > ROOT > > <http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b> > > 09 > > Sep 2021 18:01:47 > > ERROR > > <http://cm01.cs.la1.clx.corp:8080/client/#/event/39145528-236e-4724-ac59-e3f99f61c624> > > SNAPSHOT.CREATE Completed Error while taking snapshot admin > > <http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > admin > > <http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > ROOT > > <http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b> > > 09 > > Sep 2021 17:53:16 > > ERROR > > <http://cm01.cs.la1.clx.corp:8080/client/#/event/d830de84-d9da-4756-92e4-1c03a9d39f51> > > SNAPSHOT.CREATE Completed Error while taking snapshot admin > > <http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > admin > > <http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b> > > ROOT > > <http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b> > > 09 > > Sep 2021 17:45:06 > > > > I’m not really sure where to look to figured out what went wrong. I have > > VMs servinig from both NFS and Ceph with no issues. I’m able to migrate a > > volume from NFS to Ceph without any issues but snapshots fail. > > > > CentOS 8. > > > > Thanks > > -jeremy > > > > > > > > > > > > > > signature.asc Description: PGP signature
Troubleshooting failed volume snapshot?
I’m trying to figure out how to debug a snapshot failure. I’m trying to snapshot a root volume and after some time, the process fails but it doesn’t really give any information. Manager host is running cs 4.15.0.0 VM Hosts are running agent version 4.15.1.0 Primary storage is both Ceph and NFS Secondary is just NFS ERROR (http://cm01.cs.la1.clx.corp:8080/client/#/event/2e996766-1e53-4d3e-840c-ef9c2759c777) VOLUME.MIGRATE Completed Error while migrating volume admin (http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b) admin (http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b) ROOT (http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b) 09 Sep 2021 18:01:47 ERROR (http://cm01.cs.la1.clx.corp:8080/client/#/event/39145528-236e-4724-ac59-e3f99f61c624) SNAPSHOT.CREATE Completed Error while taking snapshot admin (http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b) admin (http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b) ROOT (http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b) 09 Sep 2021 17:53:16 ERROR (http://cm01.cs.la1.clx.corp:8080/client/#/event/d830de84-d9da-4756-92e4-1c03a9d39f51) SNAPSHOT.CREATE Completed Error while taking snapshot admin (http://cm01.cs.la1.clx.corp:8080/client/#/accountuser?username=admin=daa316d7-d004-11eb-a46a-47e295469f4b) admin (http://cm01.cs.la1.clx.corp:8080/client/#/account?name=admin=daa316d7-d004-11eb-a46a-47e295469f4b) ROOT (http://cm01.cs.la1.clx.corp:8080/client/#/domain/daa316d7-d004-11eb-a46a-47e295469f4b) 09 Sep 2021 17:45:06 I’m not really sure where to look to figured out what went wrong. I have VMs servinig from both NFS and Ceph with no issues. I’m able to migrate a volume from NFS to Ceph without any issues but snapshots fail. CentOS 8. Thanks -jeremy signature.asc Description: PGP signature
Re: Error during live migration
I found the answer here:
https://dacat.cc/1969.html
I don’t understand why this was an issue though. First time I’ve encountered
this and I haven’t specified a UUID on any other nodes.
-jeremy
> On Wednesday, Sep 01, 2021 at 7:54 PM, Jeremy Hansen (mailto:jer...@skidrow.la)> wrote:
> I noticed trying to move a VM between specific hosts results in this error:
>
> Exception during migrate: org.libvirt.LibvirtException: internal error:
> Attempt to migrate guest to the same host 03000200-0400-0500-0006-000700080009
>
> 2021-09-01 19:51:44,166 - Executing command: get_rule_logs_for_vms
> 2021-09-01 19:52:44,166 - Executing command: get_rule_logs_for_vms
> 2021-09-01 19:52:51,998 - Executing command: destroy_network_rules_for_vm
> 2021-09-01 19:52:51,999 - iptables-save | awk
> '/BF(.*)physdev-is-bridged(.*)i-4-27-def/ { sub(/-A/, "-D", $1) ; print }'
> 2021-09-01 19:52:52,006 - ip6tables-save | awk
> '/BF(.*)physdev-is-bridged(.*)i-4-27-def/ { sub(/-A/, "-D", $1) ; print }'
> 2021-09-01 19:52:52,012 - ebtables -t nat -L PREROUTING | grep i-4-27-VM
> 2021-09-01 19:52:52,020 - Command exited non-zero: ebtables -t nat -L
> PREROUTING | grep i-4-27-VM
> Traceback (most recent call last):
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 53, in execute
> return check_output(cmd, shell=True).decode()
> File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
> **kwargs).stdout
> File "/usr/lib64/python3.6/subprocess.py", line 438, in run
> output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'ebtables -t nat -L PREROUTING | grep
> i-4-27-VM' returned non-zero exit status 1.
> 2021-09-01 19:52:52,022 - ebtables -t nat -L POSTROUTING | grep i-4-27-VM
> 2021-09-01 19:52:52,030 - Command exited non-zero: ebtables -t nat -L
> POSTROUTING | grep i-4-27-VM
> Traceback (most recent call last):
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 53, in execute
> return check_output(cmd, shell=True).decode()
> File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
> **kwargs).stdout
> File "/usr/lib64/python3.6/subprocess.py", line 438, in run
> output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'ebtables -t nat -L POSTROUTING | grep
> i-4-27-VM' returned non-zero exit status 1.
> 2021-09-01 19:52:52,030 - ebtables -t nat -F i-4-27-VM-in
> 2021-09-01 19:52:52,037 - Command exited non-zero: ebtables -t nat -F
> i-4-27-VM-in
> Traceback (most recent call last):
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 53, in execute
> return check_output(cmd, shell=True).decode()
> File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
> **kwargs).stdout
> File "/usr/lib64/python3.6/subprocess.py", line 438, in run
> output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-in'
> returned non-zero exit status 1.
> 2021-09-01 19:52:52,037 - Ignoring failure to delete ebtables chain for vm
> i-4-27-VM
> 2021-09-01 19:52:52,037 - ebtables -t nat -F i-4-27-VM-out
> 2021-09-01 19:52:52,044 - Command exited non-zero: ebtables -t nat -F
> i-4-27-VM-out
> Traceback (most recent call last):
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 53, in execute
> return check_output(cmd, shell=True).decode()
> File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
> **kwargs).stdout
> File "/usr/lib64/python3.6/subprocess.py", line 438, in run
> output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-out'
> returned non-zero exit status 1.
> 2021-09-01 19:52:52,045 - Ignoring failure to delete ebtables chain for vm
> i-4-27-VM
> 2021-09-01 19:52:52,045 - ebtables -t nat -F i-4-27-VM-in-ips
> 2021-09-01 19:52:52,051 - Command exited non-zero: ebtables -t nat -F
> i-4-27-VM-in-ips
> Traceback (most recent call last):
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 53, in execute
> return check_output(cmd, shell=True).decode()
> File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
> **kwargs).stdout
> File "/usr/lib64/python3.6/subprocess.py", line 438, in run
> output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-in-ips'
> returned non-zero exit status 1.
> 2021-09-01 19:52:52,052 - Ignoring failure to delete ebtables chain for vm
> i-4-27-VM
> 2021-09-01 19:52:52,052 - ebtables -t nat -F i-4-27-VM-
Error during live migration
I noticed trying to move a VM between specific hosts results in this error:
Exception during migrate: org.libvirt.LibvirtException: internal error: Attempt
to migrate guest to the same host 03000200-0400-0500-0006-000700080009
2021-09-01 19:51:44,166 - Executing command: get_rule_logs_for_vms
2021-09-01 19:52:44,166 - Executing command: get_rule_logs_for_vms
2021-09-01 19:52:51,998 - Executing command: destroy_network_rules_for_vm
2021-09-01 19:52:51,999 - iptables-save | awk
'/BF(.*)physdev-is-bridged(.*)i-4-27-def/ { sub(/-A/, "-D", $1) ; print }'
2021-09-01 19:52:52,006 - ip6tables-save | awk
'/BF(.*)physdev-is-bridged(.*)i-4-27-def/ { sub(/-A/, "-D", $1) ; print }'
2021-09-01 19:52:52,012 - ebtables -t nat -L PREROUTING | grep i-4-27-VM
2021-09-01 19:52:52,020 - Command exited non-zero: ebtables -t nat -L
PREROUTING | grep i-4-27-VM
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line
53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ebtables -t nat -L PREROUTING | grep
i-4-27-VM' returned non-zero exit status 1.
2021-09-01 19:52:52,022 - ebtables -t nat -L POSTROUTING | grep i-4-27-VM
2021-09-01 19:52:52,030 - Command exited non-zero: ebtables -t nat -L
POSTROUTING | grep i-4-27-VM
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line
53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ebtables -t nat -L POSTROUTING | grep
i-4-27-VM' returned non-zero exit status 1.
2021-09-01 19:52:52,030 - ebtables -t nat -F i-4-27-VM-in
2021-09-01 19:52:52,037 - Command exited non-zero: ebtables -t nat -F
i-4-27-VM-in
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line
53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-in'
returned non-zero exit status 1.
2021-09-01 19:52:52,037 - Ignoring failure to delete ebtables chain for vm
i-4-27-VM
2021-09-01 19:52:52,037 - ebtables -t nat -F i-4-27-VM-out
2021-09-01 19:52:52,044 - Command exited non-zero: ebtables -t nat -F
i-4-27-VM-out
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line
53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-out'
returned non-zero exit status 1.
2021-09-01 19:52:52,045 - Ignoring failure to delete ebtables chain for vm
i-4-27-VM
2021-09-01 19:52:52,045 - ebtables -t nat -F i-4-27-VM-in-ips
2021-09-01 19:52:52,051 - Command exited non-zero: ebtables -t nat -F
i-4-27-VM-in-ips
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line
53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-in-ips'
returned non-zero exit status 1.
2021-09-01 19:52:52,052 - Ignoring failure to delete ebtables chain for vm
i-4-27-VM
2021-09-01 19:52:52,052 - ebtables -t nat -F i-4-27-VM-out-ips
2021-09-01 19:52:52,058 - Command exited non-zero: ebtables -t nat -F
i-4-27-VM-out-ips
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line
53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ebtables -t nat -F i-4-27-VM-out-ips'
returned non-zero exit status 1.
2021-09-01 19:52:52,059 - Ignoring failure to delete ebtables chain for vm
i-4-27-VM
2021-09-01 19:52:52,059 - ebtables -t nat -F i-4-27-VM-in-src
2021-09-01 19:52:52,065 - Command exited non-zero: ebtables -t nat -F
i-4-27-VM-in-src
Traceback (most recent call last):
File
Ethernet issues with CephFS client mount on a CS instance
I’m going to also post this to the Ceph list since it seems to only happen when I have a cephfs volume mounted from a cloudstack instance. Attempting to rsync a large file to the Ceph volume, the instance becomes unresponsive at the network level. It eventually returns but it will continually drop offline as the file copies. Dmesg shows this: [ 7144.888744] e1000e :00:19.0 eno1: Detected Hardware Unit Hang: TDH <80> TDT next_to_use next_to_clean <7f> buffer_info[next_to_clean]: time_stamp <100686d46> next_to_watch <80> jiffies <100687140> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <3800> PHY Extended Status <3000> PCI Status <10> [ 7146.872563] e1000e :00:19.0 eno1: Detected Hardware Unit Hang: TDH <80> TDT next_to_use next_to_clean <7f> buffer_info[next_to_clean]: time_stamp <100686d46> next_to_watch <80> jiffies <100687900> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <3800> PHY Extended Status <3000> PCI Status <10> [ 7148.856703] e1000e :00:19.0 eno1: Detected Hardware Unit Hang: TDH <80> TDT next_to_use next_to_clean <7f> buffer_info[next_to_clean]: time_stamp <100686d46> next_to_watch <80> jiffies <1006880c0> next_to_watch.status <0> MAC Status <80083> PHY Status <796d> PHY 1000BASE-T Status <3800> PHY Extended Status <3000> PCI Status <10> [ 7150.199756] e1000e :00:19.0 eno1: Reset adapter unexpectedly The host machine: System Information Manufacturer: Dell Inc. Product Name: OptiPlex 990 Running CentOS 8.4. I also see the same error on another host of a different hw type: Manufacturer: Hewlett-Packard Product Name: HP Compaq 8200 Elite SFF PC but both are using e1000 drivers. I upgraded the kernel to 5.13.x and I thought this fixed the issue, but now I see the error again. Migrating the instance to a bigger server class machine (also e1000e, old Rackable system) where I have a bigger pipe via bonding, I don’t seem to have the issue. Just curious if this could be a known bug with e1000e and if there is any kind of work around. Thanks -jeremy signature.asc Description: PGP signature
Re: User certificate - Cloudmonkey API keys?
Beautiful. Thank you so much! Exactly what I needed. So what is the user certificates used for? -jeremy > On Jun 25, 2021, at 3:59 AM, Slavka Peleva > wrote: > > Hi Jeremy, > > As far as I understand, from your second screenshot get into the View Users > -> required user-> user menu and there is an option to generate keys > > Best regards, > Slavka > > On Fri, Jun 25, 2021 at 12:45 PM Jeremy Hansen <mailto:jer...@skidrow.la>> wrote: > It seems to have a problem when I use a certificate with a passphrase. Using > -nodes seems to work. I don’t understand though since the form in the > interface has a place to put your passphrase. > > What I’m trying to do here is use Cloudmonkey with api keys, but I see > nothing in Cloudstack to generate these keys: > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI#CloudStackcloudmonkeyCLI-Gettingstarted > > <https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI#CloudStackcloudmonkeyCLI-Gettingstarted> > > Maybe this is referencing an older version of Cloudstack? Nothing in the > users section from what I can see mentions anything about generating api keys. > > > > > > > I see nothing about generating api keys. > > -jeremy > > >> On Jun 25, 2021, at 12:30 AM, Daan Hoogland > <mailto:daan.hoogl...@gmail.com>> wrote: >> >> Jeremy, >> It seems to me you are creating a certificate request with that command and >> not a certificate. What file(s) did you try to upload? >> Google "creating keys and certificates" to get some clues about how to >> create those. >> regards, >> >> On Thu, Jun 24, 2021 at 9:57 AM Jeremy Hansen > <mailto:jer...@skidrow.la>> wrote: >> I’m trying to understand how to generate a user certificate. I can’t seem >> to find very much information. >> >> >> >> >> I tried: >> >> openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 >> >> but the output just results in an error: >> >> >> >> Any clues? >> >> Thanks >> -jeremy >> >> >> >> -- >> Daan > signature.asc Description: Message signed with OpenPGP
User certificate
I’m trying to understand how to generate a user certificate. I can’t seem to find very much information. I tried: openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 but the output just results in an error: Any clues? Thanks -jeremy signature.asc Description: Message signed with OpenPGP
Re: HA State: Ineligible
Just seems strange. If I have centralized storage and I can migrate live vm’s, why wouldn’t I be able to take the risk to migrate a dead vm guest, as in, just start it on another available host and start it and consider it migrated. If the original vm host happens to come back up and the instance has already migrated, just leave it alone. Seems like that would go a long way for high availability. I assume i/o fencing becomes an issue. I’d prefer some kind of hook so if a VM host seems dead beyond, say, 5 minutes, i/o fence it by triggering a power off from the PDU and bring up the VM guests somewhere else. -jeremy > On Jun 18, 2021, at 2:42 AM, Daan Hoogland wrote: > > :D > I'm sorry, when your bike is broken you have to walk. detection if the > exact situation is the issue here, good luck in your search! > > On Fri, Jun 18, 2021 at 11:31 AM Jeremy Hansen wrote: > >> I guess I’m looking for a recovery scenario where the dead vm host is not >> coming back, failed disk, caught on fire, and a reboot isn’t going to help. >> >> Thanks >> >>> On Jun 18, 2021, at 1:41 AM, Daan Hoogland >> wrote: >>> >>> Jemery, >>> If you don't have IPMI then ACS can not know for sure that the VM won't >>> come back. If it comes back the VM would be running twice and this must >> be >>> prevented at all costs. Maybe I am missing some functionality, and >> someone >>> else can give additional options. >>> >>>> On Fri, Jun 18, 2021 at 10:21 AM Jeremy Hansen >> wrote: >>>> >>>> I pasted that from the documentation. My end goal is if a VM host dies >>>> completely and I’m not available to fix it, I would like the VM guests >> that >>>> were running on the failed host to automatically migrate to an >> available VM >>>> host so the guest instances continue to run. Perhaps that’s not how it >>>> works. The hosts I’m using for testing do not have any kind of IPMI >>>> supported out of band management. They do have network enabled PDUs but >>>> let’s just say the VM host is gone completely. How do I get the VM >> guests >>>> that were running on the failed host back up and running without my >>>> intervention? I guess I wrongly assumed Cloudstack would handle this >> case >>>> by just starting the VMs on another available host machine after some >> kind >>>> of failed heartbeat threshold. >>>> >>>> Thanks >>>> -jeremy >>>> >>>>> On Jun 18, 2021, at 1:09 AM, Daan Hoogland >>>> wrote: >>>>> >>>>> Jeremy, >>>>> I don't fully understand your question. You say you are interested in >>>>> HostHA specifically but then you ask about restarting VMs when a host >>>> dies. >>>>> This would not be safe as we can't be sure a host really dies unless >> you >>>>> have HostHA enabled. Consequently you can't guarantee the VM won't >>>> suddenly >>>>> re-apear when the host is seen running again. So keep these things >>>>> separated. >>>>> HostHA is for rebooting suspect hosts, not for moving VMs around. I am >>>> not >>>>> aware of the connection between the two, that you seem to look for. >>>>> >>>>> >>>>> On Fri, Jun 11, 2021 at 11:03 AM Jeremy Hansen >>>> wrote: >>>>> >>>>>> >>>>>> I’m trying to play with HA. I’ve enabled it via the interface but the >>>> HA >>>>>> state is labeled as Ineligible. >>>>>> >>>>>> I’m specifically interested in this: >>>>>> >>>>>> HA for Hosts >>>>>> >>>>>> The user can specify a virtual machine as HA-enabled. By default, all >>>>>> virtual router VMs and Elastic Load Balancing VMs are automatically >>>>>> configured as HA-enabled. When an HA-enabled VM crashes, CloudStack >>>> detects >>>>>> the crash and restarts the VM automatically within the same >> Availability >>>>>> Zone. HA is never performed across different Availability Zones. >>>> CloudStack >>>>>> has a conservative policy towards restarting VMs and ensures that >> there >>>>>> will never be two instances of the same VM running at the same time. >> The >>>>>> Management Server attempts to start the VM on another Host in the same >>>>>> cluster. >>>>>> >>>>>> >>>>>> My assumption is if a VM Host dies, whatever guests that were running >> on >>>>>> that host would automatically move to an available VM host. Maybe I’m >>>>>> misinterpreting. >>>>>> >>>>>> Thanks >>>>>> -jeremy >>>>>> >>>>> >>>>> >>>>> -- >>>>> Daan >>>> >>>> >>> >>> -- >>> Daan >> >> > > -- > Daan signature.asc Description: Message signed with OpenPGP
Re: HA State: Ineligible
I guess I’m looking for a recovery scenario where the dead vm host is not coming back, failed disk, caught on fire, and a reboot isn’t going to help. Thanks > On Jun 18, 2021, at 1:41 AM, Daan Hoogland wrote: > > Jemery, > If you don't have IPMI then ACS can not know for sure that the VM won't > come back. If it comes back the VM would be running twice and this must be > prevented at all costs. Maybe I am missing some functionality, and someone > else can give additional options. > >> On Fri, Jun 18, 2021 at 10:21 AM Jeremy Hansen wrote: >> >> I pasted that from the documentation. My end goal is if a VM host dies >> completely and I’m not available to fix it, I would like the VM guests that >> were running on the failed host to automatically migrate to an available VM >> host so the guest instances continue to run. Perhaps that’s not how it >> works. The hosts I’m using for testing do not have any kind of IPMI >> supported out of band management. They do have network enabled PDUs but >> let’s just say the VM host is gone completely. How do I get the VM guests >> that were running on the failed host back up and running without my >> intervention? I guess I wrongly assumed Cloudstack would handle this case >> by just starting the VMs on another available host machine after some kind >> of failed heartbeat threshold. >> >> Thanks >> -jeremy >> >>> On Jun 18, 2021, at 1:09 AM, Daan Hoogland >> wrote: >>> >>> Jeremy, >>> I don't fully understand your question. You say you are interested in >>> HostHA specifically but then you ask about restarting VMs when a host >> dies. >>> This would not be safe as we can't be sure a host really dies unless you >>> have HostHA enabled. Consequently you can't guarantee the VM won't >> suddenly >>> re-apear when the host is seen running again. So keep these things >>> separated. >>> HostHA is for rebooting suspect hosts, not for moving VMs around. I am >> not >>> aware of the connection between the two, that you seem to look for. >>> >>> >>> On Fri, Jun 11, 2021 at 11:03 AM Jeremy Hansen >> wrote: >>> >>>> >>>> I’m trying to play with HA. I’ve enabled it via the interface but the >> HA >>>> state is labeled as Ineligible. >>>> >>>> I’m specifically interested in this: >>>> >>>> HA for Hosts >>>> >>>> The user can specify a virtual machine as HA-enabled. By default, all >>>> virtual router VMs and Elastic Load Balancing VMs are automatically >>>> configured as HA-enabled. When an HA-enabled VM crashes, CloudStack >> detects >>>> the crash and restarts the VM automatically within the same Availability >>>> Zone. HA is never performed across different Availability Zones. >> CloudStack >>>> has a conservative policy towards restarting VMs and ensures that there >>>> will never be two instances of the same VM running at the same time. The >>>> Management Server attempts to start the VM on another Host in the same >>>> cluster. >>>> >>>> >>>> My assumption is if a VM Host dies, whatever guests that were running on >>>> that host would automatically move to an available VM host. Maybe I’m >>>> misinterpreting. >>>> >>>> Thanks >>>> -jeremy >>>> >>> >>> >>> -- >>> Daan >> >> > > -- > Daan
Re: HA State: Ineligible
I pasted that from the documentation. My end goal is if a VM host dies completely and I’m not available to fix it, I would like the VM guests that were running on the failed host to automatically migrate to an available VM host so the guest instances continue to run. Perhaps that’s not how it works. The hosts I’m using for testing do not have any kind of IPMI supported out of band management. They do have network enabled PDUs but let’s just say the VM host is gone completely. How do I get the VM guests that were running on the failed host back up and running without my intervention? I guess I wrongly assumed Cloudstack would handle this case by just starting the VMs on another available host machine after some kind of failed heartbeat threshold. Thanks -jeremy > On Jun 18, 2021, at 1:09 AM, Daan Hoogland wrote: > > Jeremy, > I don't fully understand your question. You say you are interested in > HostHA specifically but then you ask about restarting VMs when a host dies. > This would not be safe as we can't be sure a host really dies unless you > have HostHA enabled. Consequently you can't guarantee the VM won't suddenly > re-apear when the host is seen running again. So keep these things > separated. > HostHA is for rebooting suspect hosts, not for moving VMs around. I am not > aware of the connection between the two, that you seem to look for. > > > On Fri, Jun 11, 2021 at 11:03 AM Jeremy Hansen wrote: > >> >> I’m trying to play with HA. I’ve enabled it via the interface but the HA >> state is labeled as Ineligible. >> >> I’m specifically interested in this: >> >> HA for Hosts >> >> The user can specify a virtual machine as HA-enabled. By default, all >> virtual router VMs and Elastic Load Balancing VMs are automatically >> configured as HA-enabled. When an HA-enabled VM crashes, CloudStack detects >> the crash and restarts the VM automatically within the same Availability >> Zone. HA is never performed across different Availability Zones. CloudStack >> has a conservative policy towards restarting VMs and ensures that there >> will never be two instances of the same VM running at the same time. The >> Management Server attempts to start the VM on another Host in the same >> cluster. >> >> >> My assumption is if a VM Host dies, whatever guests that were running on >> that host would automatically move to an available VM host. Maybe I’m >> misinterpreting. >> >> Thanks >> -jeremy >> > > > -- > Daan signature.asc Description: Message signed with OpenPGP
Failure creating a template from a snapshot
I see this error: Create template (GSA Security Scanner) Failed to copy snapshot:java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback Any clues on this? Thanks -jeremy signature.asc Description: Message signed with OpenPGP
Re: Error trying to create a volume snapshot
Also, I noticed when I initiate the snapshot from the Cloudstack side, an image does get allocated from Ceph’s perspective but then it ultimately fails but the snapshot remains on the ceph side. -jeremy > On Jun 15, 2021, at 5:52 PM, Jeremy Hansen wrote: > > Thank you. Creating the snapshot on the Ceph side seems to be no problem and > pretty much instantaneous. It’s just odd since I’m able to snapshot other VM > root volumes also residing on the ceph rbd pool. > > -jeremy > > >> On Jun 15, 2021, at 3:24 AM, Andrija Panic wrote: >> >> I would focus on the actual qemu-img error (i.e. nothing to do with >> CloudStack, but something to do with Ceph+RBD - try to do internal Ceph >> snapshot, see if the image is corrupted or not, etc, etc): >> >> QemuImgException: qemu-img: error while writing sector 7921663: >> Input/output errorqemu-img: Failed to flush the L2 table cache: >> Input/output errorqemu-img: Failed to flush the refcount block cache: >> Input/output error >> >> Perhaps someone else have seen something similar. >> >> >> On Tue, 15 Jun 2021 at 09:32, Jeremy Hansen wrote: >> >>> >>> For some reason a particular snapshot I’m trying to create for a volume >>> fails and I see this error: >>> >>> 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] >>> (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) >>> (logid:da060faf) Take snapshot: 49 failed >>> com.cloud.utils.exception.CloudRuntimeException: >>> org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while >>> writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 >>> table cache: Input/output errorqemu-img: Failed to flush the refcount block >>> cache: Input/output error >>> at >>> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) >>> at >>> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) >>> at >>> com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) >>> at >>> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>> at >>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>> at >>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) >>> at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) >>> at >>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) >>> at >>> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) >>> at >>> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>> at >>> com.cloud.storage.VolumeApiServiceImpl.han
Re: Error trying to create a volume snapshot
Thank you. Creating the snapshot on the Ceph side seems to be no problem and pretty much instantaneous. It’s just odd since I’m able to snapshot other VM root volumes also residing on the ceph rbd pool. -jeremy > On Jun 15, 2021, at 3:24 AM, Andrija Panic wrote: > > I would focus on the actual qemu-img error (i.e. nothing to do with > CloudStack, but something to do with Ceph+RBD - try to do internal Ceph > snapshot, see if the image is corrupted or not, etc, etc): > > QemuImgException: qemu-img: error while writing sector 7921663: > Input/output errorqemu-img: Failed to flush the L2 table cache: > Input/output errorqemu-img: Failed to flush the refcount block cache: > Input/output error > > Perhaps someone else have seen something similar. > > > On Tue, 15 Jun 2021 at 09:32, Jeremy Hansen wrote: > >> >> For some reason a particular snapshot I’m trying to create for a volume >> fails and I see this error: >> >> 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] >> (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) >> (logid:da060faf) Take snapshot: 49 failed >> com.cloud.utils.exception.CloudRuntimeException: >> org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while >> writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 >> table cache: Input/output errorqemu-img: Failed to flush the refcount block >> cache: Input/output error >>at >> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) >>at >> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) >>at >> com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) >>at >> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>at >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) >>at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) >>at >> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) >>at >> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) >>at >> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>at >> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>at >> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3633) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>at >> org
Re: Rebooted and now I see unable to find storage pool
Thank you. Looks like a good document. -jeremy > On Jun 14, 2021, at 3:33 PM, Andrija Panic wrote: > > https://www.shapeblue.com/ceph-and-cloudstack-part-1/ (there are 3 parts...) > > You should also read on the client-side caching, etc (on KVM hosts). > > Best, > > On Fri, 11 Jun 2021 at 14:16, Jeremy Hansen wrote: > >> Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This >> fixed my issue. >> >> -jeremy >> >>> On Jun 10, 2021, at 6:50 PM, Jeremy Hansen wrote: >>> >>> 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage >> pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does >> the pool 'rbd' exist?: No such file or directory >>> 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD >> storage pool, cleaning up the libvirt secret >>> >>> I should mention that I’ve defined a Ceph RBD primary storage volume. >> Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to >> understand what happened here as I plan on using Ceph RBD as my primary >> storage. >>> >>> Thanks >>> -jeremy >>> >>> >>>> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: >>>> >>>> >>>> I removed all of my VMs and all volumes. I rebooted all the servers >> involved in my cluster and now I see this: >>>> >>>> 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] >> (agentRequest-Handler-2:null) (logid:4f4da278) Caught: >>>> com.cloud.utils.exception.CloudRuntimeException: Failed to create >> storage pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab >>>> at >> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) >>>> at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) >>>> at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) >>>> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) >>>> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) >>>> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >>>> at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) >>>> at com.cloud.agent.Agent.processRequest(Agent.java:661) >>>> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) >>>> at com.cloud.utils.nio.Task.call(Task.java:83) >>>> at com.cloud.utils.nio.Task.call(Task.java:29) >>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>>> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>>> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>>> at java.base/java.lang.Thread.run(Thread.java:829) >>>> >>>> and too of my VM hosts are unable to connect. >>>> >>>> How do I resolve this situation? How did I lose a storage pool? >>>> >>>> Thanks >>>> -jeremy >>>> >>>> >>> >> >> > > -- > > Andrija Panić signature.asc Description: Message signed with OpenPGP
Error trying to create a volume snapshot
For some reason a particular snapshot I’m trying to create for a volume fails and I see this error: 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) (logid:da060faf) Take snapshot: 49 failed com.cloud.utils.exception.CloudRuntimeException: org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 table cache: Input/output errorqemu-img: Failed to flush the refcount block cache: Input/output error at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3633) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy215.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at
Re: Rebooted and now I see unable to find storage pool
Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This fixed my issue. -jeremy > On Jun 10, 2021, at 6:50 PM, Jeremy Hansen wrote: > > 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage > pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the > pool 'rbd' exist?: No such file or directory > 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD > storage pool, cleaning up the libvirt secret > > I should mention that I’ve defined a Ceph RBD primary storage volume. > Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to > understand what happened here as I plan on using Ceph RBD as my primary > storage. > > Thanks > -jeremy > > >> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: >> >> >> I removed all of my VMs and all volumes. I rebooted all the servers >> involved in my cluster and now I see this: >> >> 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] >> (agentRequest-Handler-2:null) (logid:4f4da278) Caught: >> com.cloud.utils.exception.CloudRuntimeException: Failed to create storage >> pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab >>at >> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) >>at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) >>at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) >>at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) >>at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) >>at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) >>at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) >>at com.cloud.agent.Agent.processRequest(Agent.java:661) >>at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) >>at com.cloud.utils.nio.Task.call(Task.java:83) >>at com.cloud.utils.nio.Task.call(Task.java:29) >>at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>at java.base/java.lang.Thread.run(Thread.java:829) >> >> and too of my VM hosts are unable to connect. >> >> How do I resolve this situation? How did I lose a storage pool? >> >> Thanks >> -jeremy >> >> >
HA State: Ineligible
I’m trying to play with HA. I’ve enabled it via the interface but the HA state is labeled as Ineligible. I’m specifically interested in this: HA for Hosts The user can specify a virtual machine as HA-enabled. By default, all virtual router VMs and Elastic Load Balancing VMs are automatically configured as HA-enabled. When an HA-enabled VM crashes, CloudStack detects the crash and restarts the VM automatically within the same Availability Zone. HA is never performed across different Availability Zones. CloudStack has a conservative policy towards restarting VMs and ensures that there will never be two instances of the same VM running at the same time. The Management Server attempts to start the VM on another Host in the same cluster. My assumption is if a VM Host dies, whatever guests that were running on that host would automatically move to an available VM host. Maybe I’m misinterpreting. Thanks -jeremy signature.asc Description: Message signed with OpenPGP
Re: Rebooted and now I see unable to find storage pool
2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'rbd' exist?: No such file or directory 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD storage pool, cleaning up the libvirt secret I should mention that I’ve defined a Ceph RBD primary storage volume. Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to understand what happened here as I plan on using Ceph RBD as my primary storage. Thanks -jeremy > On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: > > > I removed all of my VMs and all volumes. I rebooted all the servers involved > in my cluster and now I see this: > > 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] > (agentRequest-Handler-2:null) (logid:4f4da278) Caught: > com.cloud.utils.exception.CloudRuntimeException: Failed to create storage > pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab > at > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) > at com.cloud.agent.Agent.processRequest(Agent.java:661) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > > and too of my VM hosts are unable to connect. > > How do I resolve this situation? How did I lose a storage pool? > > Thanks > -jeremy > > signature.asc Description: Message signed with OpenPGP
Rebooted and now I see unable to find storage pool
I removed all of my VMs and all volumes. I rebooted all the servers involved in my cluster and now I see this: 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:4f4da278) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) and too of my VM hosts are unable to connect. How do I resolve this situation? How did I lose a storage pool? Thanks -jeremy signature.asc Description: Message signed with OpenPGP
Re: Alter Shared Guest Network?
Thanks. I’ll take a look table. -jeremy > On Jun 10, 2021, at 6:57 AM, Yordan Kostov wrote: > > Hello Jeremy, > >Once a shared network with DHCP offering is created the IPs fitting into > the defined range are created in table called "user_ip_address". >They are created one by one so if range between x.x.x.x.11 and x.x.x.210 > is created this will add 200 entries. So if you want to expand that you need > to add more entries manually, which is a bit unfortunate. > > Best regards, > Jordan > > -Original Message- > From: Jeremy Hansen > Sent: Thursday, June 10, 2021 12:12 AM > To: users@cloudstack.apache.org > Subject: Re: Alter Shared Guest Network? > > > [X] This message came from outside your organization > > >> On Jun 9, 2021, at 1:39 PM, Wido den Hollander wrote: >> >> >> >>>> On 6/9/21 3:55 PM, Jeremy Hansen wrote: >>> When I created my shared network config, I specified too narrow of an IP >>> range. >>> >>> I can’t seem to figure out how to alter this config via the web interface. >>> Is this possible? >>> >> >> Not via de UI nor API. You will need to hack this in the database. Or >> remove the network and create it again. But this is only possible if >> there are no VMs in the network. >> >> Wido > > Thanks, recreating it seems like the easiest option since I’m only in testing > phase right now, but I’m curious what it would take to alter tables to fix > this. Any clues as to what tables/fields would need to be updated? > >> >>> -jeremy >>> >
Re: Alter Shared Guest Network?
> On Jun 9, 2021, at 1:39 PM, Wido den Hollander wrote: > > > >> On 6/9/21 3:55 PM, Jeremy Hansen wrote: >> When I created my shared network config, I specified too narrow of an IP >> range. >> >> I can’t seem to figure out how to alter this config via the web interface. >> Is this possible? >> > > Not via de UI nor API. You will need to hack this in the database. Or > remove the network and create it again. But this is only possible if > there are no VMs in the network. > > Wido Thanks, recreating it seems like the easiest option since I’m only in testing phase right now, but I’m curious what it would take to alter tables to fix this. Any clues as to what tables/fields would need to be updated? > >> -jeremy >>
Alter Shared Guest Network?
When I created my shared network config, I specified too narrow of an IP range. I can’t seem to figure out how to alter this config via the web interface. Is this possible? -jeremy
