Issues migrating primary storage

[Jeremy Hansen]

I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB 
volumes. These inevitably times out. I extended these configs:

copy.volume.wait=72000
job.cancel.threshold.minutes=480
job.expire.minutes=1440

This helped with smaller volumes but large once still eventually fail.

2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) 
(logid:) Seq 1-5583619113009291196: Processing: { Ans: , MgmtId: 
20558852646968, via: 1, Ver: v1, Flags: 10, 
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
 Failed to copy 
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e 
to 
b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
 }

2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl] 
(Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b) 
(logid:d7d98b81) Failed to migrate volume
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is 
unreachable: Volume 
[{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}] 
migration failed due to [com.cloud.utils.exception.CloudRuntimeException: 
Failed to copy 
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e 
to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
at 
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
at 
com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
at 
com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at 
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.c

Re: new PMC member Harikrishna Patnala

[Wei ZHOU]

Congratulations Hari !

-Wei

On Mon, 15 Jan 2024 at 10:25, Daan Hoogland  wrote:

> users and dev,
>
> The PMC have invited Harikrishna to join their ranks and he has
> gracefully accepted. Please join me in congratulating Hari.
>
> --
> Daan
>

Seeking Insights on Cloudstack Implementation with VXLAN, BGP, and IPV6

[Hunter Yap]

Hey Wido and everyone in the Community,

Hope you're all doing great. We're setting up Cloudstack and stumbled upon
your cool videos about VXLAN, BGP, and IPV6 from 2019 and 2021. Watched
them all, including the links you shared. Noticed that similar concepts pop
up in Openstack and OpenNebula too, with a few twists.

So, we've got a few things on our mind about the setup, especially how it
fits (or doesn't) with what we're doing. You mentioned in your video to
reach out with questions, so here we are, looping you in.

Our Setup Is Where:
# Cloudstack V4.18.1 + Linstor (For SDS)
# VXLAN Enabled for Guest (Using Multicast)
# 1 Pair of Redundant Core Switch (With option to scale out)
# 1 Pair of Redundant Leaf Switch
# Our datacenter can handle up to 1,000 hypervisors, which is more than
enough for us.

Our setup isn’t huge – just right for supporting a max of 1,000 hypervisors
per datacenter. And, we're not planning to link multiple datacenters into
one big availability zone. Having said that, we're scratching our heads
over whether BGP+EVPN is the way to go for us.

Here's what we're thinking:

# BGP + EVPN doesn't add additional benefit to small/medium sized clouds.
(In our case, we only need Max 1,000 Hypervisors per Datacenter).
Implementing BGP+EVPN only increases the complexity of the setup.

# We are using IPV4. But it seems that for this BGP+EVPN to be beneficial,
we need to use IPV6. If we use back IPV4 (with BGP+EVPN), there is no
benefit than if we were to use the default VXLAN+Multicast.

# And about redundancies – doesn't look like VXLAN+BGP+EVPN offers anything
more than what VXLAN+Multicast already does.

Our Assumptions, To enable BGP + EVPN:

1. We need to instal FRRouter in each Hypervisor and form BGP Neighbor to
the Leaf Switch (so this is Hypervisor <> Leaf Switch and NOT Hypervisors
<> Hypervisor).

2. After forming neighbours with the Leaf Switch, we need to enable L2VPN
at all leaf switches.

3. When the first 2 steps above are done, this would then mean that each
Hypervisor will be as if it is sitting in its own network segment. (Usually
all hypervisors will be in 1 single network segment)

4. We run the modifyvxlan.sh provided on Github and replace it with the
default one from Cloudstack.

5. That's it, basically run Cloudstack and test to see if it works. (That's
all the steps we're aware of).

Note: Our assumption is also that this BGP setup will only affect internal
Guest Network (VXLAN) communications. Networks to the Public Internet
(VLAN) will not be using this BGP.

Would love to get your thoughts on this. Are we on the right track or
missing something? Any advice or heads-up would be great.

Regards,
Hunter Yap

Re: Apache CloudStack Deployment Issue

[Wei ZHOU]

Hi,

The public IPs should be accessible from (at least) your browser. If not,
you will not be able to access the vm console and upload templates/iso from
local.
You can register template/iso from URL instead.

-Wei

On Tue, 16 Jan 2024 at 04:40, Bharat Bhushan Saini
 wrote:

> Hi Wei,
>
>
>
> Public IP’s of the system are the IP’s of private network of the
> organisation and it is directly connected to outside through NAT.
>
> Additional Info. – What the public IP assigned to the system through
> cloudstack are not leased and didn’t pingable from the same network and
> also didn’t pingable from the hypervisor and management as well. I observed
> only one way communication from there.
>
>
>
> Thanks and Regards,
>
> Bharat Saini
>
>
>
> [image: signature_3350461708]
>
>
>
> *From: *Wei ZHOU 
> *Date: *Monday, 15 January 2024 at 1:43 PM
> *To: *users@cloudstack.apache.org 
> *Subject: *Re: Apache CloudStack Deployment Issue
>
> EXTERNAL EMAIL: Please verify the sender email address before taking any
> action, replying, clicking any link or opening any attachment.
>
>
> Hi,
>
> What are the public IPs of system VMs ? It looks like they do not work.
>
> -Wei
>
> On Mon, 15 Jan 2024 at 09:04, Bharat Bhushan Saini
>  wrote:
>
> > Hi All,
> >
> >
> >
> > Now System VM’s are up and running and ssvm.sh script also pass all the
> > parameters.
> >
> > But after uploading the certificate for vm’s through GUI, Still I am
> > unable to upload the iso and unable to open console.
> >
> >
> >
> > Note: Cloudstack and Hypervisor(KVM) both running in the proxmox infra.
> >
> >
> >
> > Thanks and Regards,
> >
> > Bharat Saini
> >
> >
> >
> > [image: signature_841501871]
> >
> >
> >
> > *From: *Bharat Bhushan Saini 
> > *Date: *Friday, 12 January 2024 at 11:23 AM
> > *To: *users@cloudstack.apache.org ,
> > ustcweiz...@gmail.com , ji...@linservers.com <
> > ji...@linservers.com>
> > *Cc: *users@cloudstack.apache.org 
> > *Subject: *Re: Apache CloudStack Deployment Issue
> >
> > EXTERNAL EMAIL: Please verify the sender email address before taking any
> > action, replying, clicking any link or opening any attachment.
> >
> >
> >
> > Hi All,
> >
> >
> >
> > VLAN and many other things was managed by proxmox infra.
> >
> > And I already follow confluence page for ssvm troubleshooting and found
> > that gateway is not pinged. I will again raise my issue as below,
> >
> > 1.  When ISO has been adding then error encountered that “There is no
> > secondary storage VM for downloading template to image store ISO-Store”.
> > 2.  When going to check vm through console it is not
> > opened in the browser.
> > 3.  When logged in the ssvm from backend, vm is not able
> > to ping google DNS and when ssvm-check.sh script run it is not able to
> ping
> > it’s gateway as well.
> >
> >
> >
> > Thanks and Regards,
> >
> > Bharat Saini
> >
> >
> >
> > [image: signature_2013149965]
> >
> >
> >
> > *From: *Jithin Raju 
> > *Date: *Friday, 12 January 2024 at 10:28 AM
> > *To: *users@cloudstack.apache.org ,
> > ustcweiz...@gmail.com , ji...@linservers.com <
> > ji...@linservers.com>
> > *Cc: *users@cloudstack.apache.org 
> > *Subject: *Re: Apache CloudStack Deployment Issue
> >
> > EXTERNAL EMAIL: Please verify the sender email address before taking any
> > action, replying, clicking any link or opening any attachment.
> >
> >
> > Hi Bharat,
> >
> > Can you check the VLAN configuration on the switch ports? Make sure the
> > SSVM vNICs are on the expected bridges?
> >
> > -Jithin
> >
> > From: Bharat Bhushan Saini 
> > Date: Thursday, 11 January 2024 at 11:38 PM
> > To: ustcweiz...@gmail.com , ji...@linservers.com
> <
> > ji...@linservers.com>
> > Cc: users@cloudstack.apache.org 
> > Subject: Re: Apache CloudStack Deployment Issue
> > Hi Wei and Jimmy,
> >
> > I already deployed cloudstack version 4.18 with 1 management server
> > included mysql and nfs in it and 1 kvm with libvirt, also both system was
> > running on the ubuntu 22.04 LTS.
> > After adding a zone, facts are below,
> >
> >   1.  Zone is enabled.
> >   2.  Host server is up and running.
> >   3.  Primary storage is add and up with nfs.
> >   4.  Secondary storage is added with nfs.
> >   5.  System VM’s of console and ssvm is up and running.
> >   6.  Default template is added Debian GNU and Centos 5.5 but Centos 5.5
> > is not in ready state.
> >
> >
> >
> > Issues encountered,
> >
> >   1.  When ISO has been adding then error encountered that “There is no
> > secondary storage VM for downloading template to image store ISO-Store”.
> >   2.  When going to check vm through console it is not opened in the
> > browser.
> >   3.  When logged in the ssvm from backend, vm is not able to ping google
> > DNS and when ssvm-check.sh script run it is not able to ping it’s gateway
> > as well.
> >
> > Kindly help out from the configuration part. I also try wireguard for
> this
> > but it is also not done yet.
> >
> > Thanks and Regards,
> > Bharat Saini
> >
> > [signa

Re: new PMC member Harikrishna Patnala

[Harikrishna Patnala]

Thank you everyone.

Regards,
Harikrishna


From: Wei ZHOU 
Sent: Tuesday, January 16, 2024 2:16:30 PM
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: new PMC member Harikrishna Patnala

Congratulations Hari !

-Wei


 

On Mon, 15 Jan 2024 at 10:25, Daan Hoogland  wrote:

> users and dev,
>
> The PMC have invited Harikrishna to join their ranks and he has
> gracefully accepted. Please join me in congratulating Hari.
>
> --
> Daan
>

Re: new PMC member Harikrishna Patnala

[Vishesh Jindal]

Congrats Hari!

From: Harikrishna Patnala 
Sent: Tuesday, January 16, 2024 2:55 PM
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: new PMC member Harikrishna Patnala

Thank you everyone.

Regards,
Harikrishna


From: Wei ZHOU 
Sent: Tuesday, January 16, 2024 2:16:30 PM
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: new PMC member Harikrishna Patnala

Congratulations Hari !

-Wei





 

On Mon, 15 Jan 2024 at 10:25, Daan Hoogland  wrote:

> users and dev,
>
> The PMC have invited Harikrishna to join their ranks and he has
> gracefully accepted. Please join me in congratulating Hari.
>
> --
> Daan
>

Re: new PMC member Harikrishna Patnala

[Pearl d'Silva]

Congratulations Hari!!

Regards,
Pearl

From: Vishesh Jindal 
Sent: January 16, 2024 5:08 AM
To: users@cloudstack.apache.org ; 
d...@cloudstack.apache.org 
Subject: Re: new PMC member Harikrishna Patnala

Congrats Hari!

From: Harikrishna Patnala 
Sent: Tuesday, January 16, 2024 2:55 PM
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: new PMC member Harikrishna Patnala

Thank you everyone.

Regards,
Harikrishna


From: Wei ZHOU 
Sent: Tuesday, January 16, 2024 2:16:30 PM
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: new PMC member Harikrishna Patnala

Congratulations Hari !

-Wei








 

On Mon, 15 Jan 2024 at 10:25, Daan Hoogland  wrote:

> users and dev,
>
> The PMC have invited Harikrishna to join their ranks and he has
> gracefully accepted. Please join me in congratulating Hari.
>
> --
> Daan
>

Backing up Cloudstack and KVM Hosts

[Granwille Strauss]

Hi Guys

Compliments of the new year! I would like to confirm what's the best way 
to backup a cloudstack and KVM server? Or what is the approach that most 
use? Do you guys make full server Image backups of the Cloudstack server 
with tools such as Veam or Acronis for example?


Then, for a KVM host where VMs use local primary storage, what's the 
recommended way to backup your KVM servers in the event of hardware 
failure? I basically want to confirm what's the best way to backup the 
dedicated KVM and Cloudstack servers in case of _hardware_ failures.


What recommended solutions work best that have been tried and tested?

--
Regards / Groete

 Granwille Strauss  // Senior Systems Admin

*e:* granwi...@namhost.com
*m:* +264 81 323 1260 
*w:* www.namhost.com 





Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy


Powered by AdSigner 


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Issues migrating primary storage

[Suresh Kumar Anaparti]

Hi Jeremy,

Can you extend with the config 'migratewait' and check.

Regards,
Suresh

On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen 
wrote:

> I have some large volumes I’m trying to migrate from NFS to Ceph/RBD.  1TB
> volumes.  These inevitably times out.  I extended these configs:
>
> copy.volume.wait=72000
> job.cancel.threshold.minutes=480
> job.expire.minutes=1440
>
> This helped with smaller volumes but large once still eventually fail.
>
> 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> Processing:  { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> to
> b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> }
>
> 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> (logid:d7d98b81) Failed to migrate volume
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> is unreachable: Volume
> [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal

Force delete/remove instances from lost host

[cristian.c]

Hello,

 

   Is there a way to force delete instances from a lost host? I still see
the VMs in CloudStack with state running/stopping. This host cannot be
recovered, I must remove this also.

 

 

 

Regards,

Cristian

CloudStack and Ceph Day - Feb 22nd, Amsterdam

[Ivet Petrova]

Hi all,

I would like to invite you all to register for the CloudStack and Ceph Day 
Netherlands 2024!
https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757

We have not done an event with the Ceph community for more than a year and I am 
very excited to meet both communities in Amsterdam.
I am sure many people from our community are either using Ceph or considering 
it as a leading storage for cloud builders.
This event will present:
- updates on Ceph
- latest 4.19 ACS release
- deep dive into the ACS integrations.

We have some great speakers from Clyso, 42on, ShapeBlue and Ampere. Final 
agenda will be announced soon!
Register now: 
https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757

Best regards,


 

Re: CloudStack and Ceph Day - Feb 22nd, Amsterdam

[Ricardo Pertuz]

Hey..

It will be streamed?


BR,

Ricardo Pertuz


On 16 Jan 2024 at 10:22 AM -0500, Ivet Petrova , 
wrote:
> Hi all,
>
> I would like to invite you all to register for the CloudStack and Ceph Day 
> Netherlands 2024!
> https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757
>
> We have not done an event with the Ceph community for more than a year and I 
> am very excited to meet both communities in Amsterdam.
> I am sure many people from our community are either using Ceph or considering 
> it as a leading storage for cloud builders.
> This event will present:
> - updates on Ceph
> - latest 4.19 ACS release
> - deep dive into the ACS integrations.
>
> We have some great speakers from Clyso, 42on, ShapeBlue and Ampere. Final 
> agenda will be announced soon!
> Register now: 
> https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757
>
> Best regards,
>
>
>
>

Re: CloudStack and Ceph Day - Feb 22nd, Amsterdam

[Ivet Petrova]

Hi,

At this stage I cannot confirm for sure.
So whoever wants to learn and see the event, it is better to be in place in 
Amsterdam.
Best regards,


 

On 16 Jan 2024, at 17:26, Ricardo Pertuz  
wrote:

Hey..

It will be streamed?


BR,

Ricardo Pertuz


On 16 Jan 2024 at 10:22 AM -0500, Ivet Petrova , 
wrote:
Hi all,

I would like to invite you all to register for the CloudStack and Ceph Day 
Netherlands 2024!
https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757

We have not done an event with the Ceph community for more than a year and I am 
very excited to meet both communities in Amsterdam.
I am sure many people from our community are either using Ceph or considering 
it as a leading storage for cloud builders.
This event will present:
- updates on Ceph
- latest 4.19 ACS release
- deep dive into the ACS integrations.

We have some great speakers from Clyso, 42on, ShapeBlue and Ampere. Final 
agenda will be announced soon!
Register now: 
https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757

Best regards,

Re: Force delete/remove instances from lost host

[Wei ZHOU]

local or shared storage ?

-Wei

On Tue, 16 Jan 2024 at 16:11,  wrote:

> Hello,
>
>
>
>Is there a way to force delete instances from a lost host? I still see
> the VMs in CloudStack with state running/stopping. This host cannot be
> recovered, I must remove this also.
>
>
>
>
>
>
>
> Regards,
>
> Cristian
>
>

Re: Force delete/remove instances from lost host

[Cristian Ciobanu]

I managed to remove, changed the status from db to stopped and then delete
from UI.

Are with local storage.

Thank you,
Cristian

On Tue, Jan 16, 2024, 17:58 Wei ZHOU  wrote:

> local or shared storage ?
>
> -Wei
>
> On Tue, 16 Jan 2024 at 16:11,  wrote:
>
> > Hello,
> >
> >
> >
> >Is there a way to force delete instances from a lost host? I still see
> > the VMs in CloudStack with state running/stopping. This host cannot be
> > recovered, I must remove this also.
> >
> >
> >
> >
> >
> >
> >
> > Regards,
> >
> > Cristian
> >
> >
>

Migrate standalone KVM to CS

[Jimmy Huybrechts]

I asked it before and I know in 4.19 there will be an option to do this easier, 
bI think I saw somewhere the release of 4.19 is set for Q3/4 (even though we 
have RC2 already, so might be sooner?) and the RC’s you can’t upgrade later on 
so other than internal testing with 4.19 we can’t use that.

If the case indeed is Q3/4 we can’t wait that long in migrating from standalone 
to CS, would then still the best case be to shut the vm, download qcow file, 
upload it to a template, build machine from it?

--
Jimmy

Re: Migrate standalone KVM to CS

[Wei ZHOU]

Hi,

if 4.19.0.0 RC2 passes, it will be officially released. In the best
scenario, it will be released in the coming two weeks.

-Wei


On Tue, 16 Jan 2024 at 17:25, Jimmy Huybrechts  wrote:

> I asked it before and I know in 4.19 there will be an option to do this
> easier, bI think I saw somewhere the release of 4.19 is set for Q3/4 (even
> though we have RC2 already, so might be sooner?) and the RC’s you can’t
> upgrade later on so other than internal testing with 4.19 we can’t use that.
>
> If the case indeed is Q3/4 we can’t wait that long in migrating from
> standalone to CS, would then still the best case be to shut the vm,
> download qcow file, upload it to a template, build machine from it?
>
> --
> Jimmy
>

Re: Migrate standalone KVM to CS

[Jimmy Huybrechts]

Hi Wei,

That is pretty good news actually, I can wait that long since being able to do 
it in 4.19 will save a whole lot of time actually in needing to do everything 
manually instead.

--
Jimmy

Van: Wei ZHOU 
Datum: dinsdag, 16 januari 2024 om 18:45
Aan: users@cloudstack.apache.org 
Onderwerp: Re: Migrate standalone KVM to CS
Hi,

if 4.19.0.0 RC2 passes, it will be officially released. In the best
scenario, it will be released in the coming two weeks.

-Wei


On Tue, 16 Jan 2024 at 17:25, Jimmy Huybrechts  wrote:

> I asked it before and I know in 4.19 there will be an option to do this
> easier, bI think I saw somewhere the release of 4.19 is set for Q3/4 (even
> though we have RC2 already, so might be sooner?) and the RC’s you can’t
> upgrade later on so other than internal testing with 4.19 we can’t use that.
>
> If the case indeed is Q3/4 we can’t wait that long in migrating from
> standalone to CS, would then still the best case be to shut the vm,
> download qcow file, upload it to a template, build machine from it?
>
> --
> Jimmy
>

MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

Hello,

 

I have a CloudStack 4.18.1 restored from a DB backup due to server failure.
The CloudStack is running; I see all the hosts with state 'Up', but it looks
like it's stuck, nothing is working. I get a similar error for all hosts:
"MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link
is closed," even if the hosts are reachable from CloudStack and vSphere,
where they are connected and manageable. I do not see any events in vSphere.

 

Any suggestions as to why CloudStack does not see that the servers are
reachable or why I get a timeout?



Thank you,

Cristian

Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[Cristian Ciobanu]

Looks like I have the same issue as I had some time ago...

"host time out after cloudstack management restore". Old topic, 2022.
 When I restored the backup on a different os, from Rocky to Ubuntu..  but
now I used the same os... Different version..

This looks like a bug. I'm trying to identify the route cause.


Regards,
Cristian

On Wed, Jan 17, 2024, 00:20  wrote:

> Hello,
>
>
>
> I have a CloudStack 4.18.1 restored from a DB backup due to server
> failure. The CloudStack is running; I see all the hosts with state 'Up',
> but it looks like it's stuck, nothing is working. I get a similar error for
> all hosts: “MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed,” even if the hosts are reachable from CloudStack
> and vSphere, where they are connected and manageable. I do not see any
> events in vSphere.
>
>
>
> Any suggestions as to why CloudStack does not see that the servers are
> reachable or why I get a timeout?
>
>
>
> Thank you,
>
> Cristian
>

Re: [VOTE] Apache CloudStack 4.19.0.0 RC2

[Nicolas Vazquez]

+1 (binding)

Tested on Vmware + KVM mixed environment: Vmware to KVM migration 
functionality, KVM unmanage/manage and VM lifecycle operations on both 
hypervisors

Regards,
Nicolas Vazquez


From: Abhishek Kumar 
Date: Monday, 15 January 2024 at 09:03
To: d...@cloudstack.apache.org , users 

Cc: PMC 
Subject: [VOTE] Apache CloudStack 4.19.0.0 RC2
Hi All,

I've created a 4.19.0.0 release (RC2), with the following artifacts up for
a vote:

Git Branch and Commit SH:
https://github.com/apache/cloudstack/tree/4.19.0.0-RC20240115T1418
Commit: 38bd4fd72bdae354c4b0f615a4861fc84d67b29a

Source release (checksums and signatures are available at the same
location):
https://dist.apache.org/repos/dist/dev/cloudstack/4.19.0.0/

PGP release keys (signed using 65518106473A09D7AF26B384A70BD2EAA74E2866):
https://dist.apache.org/repos/dist/release/cloudstack/KEYS

For testing purposes, I have uploaded the different distro packages to:
http://download.cloudstack.org/testing/4.19.0.0-RC2/

Since 4.16 the system VM template registration is no longer mandatory
before upgrading, however, it can be downloaded from here if needed:
https://download.cloudstack.org/systemvm/4.19/

The vote will be open for 72 hours.

For sanity in tallying the vote, can PMC members please be sure to indicate
"(binding)" with their vote?

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Regards,
Abhishek

 

Re: Issues migrating primary storage

[Jeremy Hansen]

Unfortunately, this didn’t seem to have an impact. Volume migration still 
eventually fails. Should I move to 4.18.1.0?

Thanks
-jeremy

> On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti 
> mailto:sureshkumar.anapa...@gmail.com)> 
> wrote:
> Hi Jeremy,
>
> Can you extend with the config 'migratewait' and check.
>
> Regards,
> Suresh
>
> On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen 
> wrote:
>
> > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > volumes. These inevitably times out. I extended these configs:
> >
> > copy.volume.wait=72000
> > job.cancel.threshold.minutes=480
> > job.expire.minutes=1440
> >
> > This helped with smaller volumes but large once still eventually fail.
> >
> > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to
> > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > }
> >
> > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > (logid:d7d98b81) Failed to migrate volume
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > is unreachable: Volume
> > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > at
> > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source)
> > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> > org

Re: Issues migrating primary storage

[Jeremy Hansen]

Upgraded to 4.18.1.0 and trying again…

-jeremy

> On Tuesday, Jan 16, 2024 at 7:08 PM, Jeremy Hansen  (mailto:jer...@skidrow.la)> wrote:
> Unfortunately, this didn’t seem to have an impact. Volume migration still 
> eventually fails. Should I move to 4.18.1.0?
>
> Thanks
> -jeremy
>
>
>
> > On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti 
> > mailto:sureshkumar.anapa...@gmail.com)> 
> > wrote:
> > Hi Jeremy,
> >
> > Can you extend with the config 'migratewait' and check.
> >
> > Regards,
> > Suresh
> >
> > On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen 
> > wrote:
> >
> > > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > > volumes. These inevitably times out. I extended these configs:
> > >
> > > copy.volume.wait=72000
> > > job.cancel.threshold.minutes=480
> > > job.expire.minutes=1440
> > >
> > > This helped with smaller volumes but large once still eventually fail.
> > >
> > > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > > to
> > > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > > }
> > >
> > > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > > (logid:d7d98b81) Failed to migrate volume
> > > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > > is unreachable: Volume
> > > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > > Failed to copy
> > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > > at
> > > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> > > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> > > Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > at
> > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > > at
> > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> > > at
> > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> > > at
> > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> > > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> > > Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> > > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> > > Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > at
> > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > > at
> > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > > at com.sun.proxy.$Proxy232.handleVmWorkJob(U

Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[Jithin Raju]

Hi Cristian,

Could you locate any stale management server record in the mshost table? If 
there is any ,specifically the one in the error ? update it as removed.

-Jithin

From: Cristian Ciobanu 
Date: Wednesday, 17 January 2024 at 4:16 AM
To: users@cloudstack.apache.org 
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed
Looks like I have the same issue as I had some time ago...

"host time out after cloudstack management restore". Old topic, 2022.
 When I restored the backup on a different os, from Rocky to Ubuntu..  but
now I used the same os... Different version..

This looks like a bug. I'm trying to identify the route cause.


Regards,
Cristian

On Wed, Jan 17, 2024, 00:20  wrote:

> Hello,
>
>
>
> I have a CloudStack 4.18.1 restored from a DB backup due to server
> failure. The CloudStack is running; I see all the hosts with state 'Up',
> but it looks like it's stuck, nothing is working. I get a similar error for
> all hosts: “MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed,” even if the hosts are reachable from CloudStack
> and vSphere, where they are connected and manageable. I do not see any
> events in vSphere.
>
>
>
> Any suggestions as to why CloudStack does not see that the servers are
> reachable or why I get a timeout?
>
>
>
> Thank you,
>
> Cristian
>

 

Re: Issues migrating primary storage

[Jeremy Hansen]

Unfortunately the upgrade didn’t help:

Resource [StoragePool:3] is unreachable: Volume 
[{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}] migration 
failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to copy 
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca 
to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].

Anything else I can try? I’m trying to move away from NFS completely.

-jeremy

> On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti 
> mailto:sureshkumar.anapa...@gmail.com)> 
> wrote:
> Hi Jeremy,
>
> Can you extend with the config 'migratewait' and check.
>
> Regards,
> Suresh
>
> On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen 
> wrote:
>
> > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> > volumes. These inevitably times out. I extended these configs:
> >
> > copy.volume.wait=72000
> > job.cancel.threshold.minutes=480
> > job.expire.minutes=1440
> >
> > This helped with smaller volumes but large once still eventually fail.
> >
> > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to
> > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> > }
> >
> > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> > (logid:d7d98b81) Failed to migrate volume
> > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> > is unreachable: Volume
> > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> > migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> > Failed to copy
> > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> > at
> > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> > Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source)
> > 

Re: Issues migrating primary storage

[Jithin Raju]

Hi Jeremy,

Have you checked the ‘wait’ parameter? Used as wait * 2 timeout.

-Jithin

From: Jeremy Hansen 
Date: Wednesday, 17 January 2024 at 12:14 PM
To: users@cloudstack.apache.org 
Subject: Re: Issues migrating primary storage
Unfortunately the upgrade didn’t help:

Resource [StoragePool:3] is unreachable: Volume 
[{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}] migration 
failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to copy 
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca 
to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].



Anything else I can try?  I’m trying to move away from NFS completely.

-jeremy




 

On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti 
mailto:sureshkumar.anapa...@gmail.com>> wrote:
Hi Jeremy,

Can you extend with the config 'migratewait' and check.

Regards,
Suresh

On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen 
wrote:


I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
volumes. These inevitably times out. I extended these configs:

copy.volume.wait=72000
job.cancel.threshold.minutes=480
job.expire.minutes=1440

This helped with smaller volumes but large once still eventually fail.

2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to copy
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
to
b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
}

2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
(Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
(logid:d7d98b81) Failed to migrate volume
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
is unreachable: Volume
[{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
Failed to copy
/mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.fra

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

Hi Jithin,

   I changed the status to down and set as removed. I still have the same 
issue...  So, I think I must create a new server with previous OS version, in 
any case not sure where is the logic.

Thank you!
Cristian 

-Original Message-
From: Jithin Raju  
Sent: Wednesday, January 17, 2024 6:16 AM
To: users@cloudstack.apache.org
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi Cristian,

Could you locate any stale management server record in the mshost table? If 
there is any ,specifically the one in the error ? update it as removed.

-Jithin

From: Cristian Ciobanu 
Date: Wednesday, 17 January 2024 at 4:16 AM
To: users@cloudstack.apache.org 
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed Looks like I have the same issue as I had some time ago...

"host time out after cloudstack management restore". Old topic, 2022.
 When I restored the backup on a different os, from Rocky to Ubuntu..  but now 
I used the same os... Different version..

This looks like a bug. I'm trying to identify the route cause.


Regards,
Cristian

On Wed, Jan 17, 2024, 00:20  wrote:

> Hello,
>
>
>
> I have a CloudStack 4.18.1 restored from a DB backup due to server 
> failure. The CloudStack is running; I see all the hosts with state 
> 'Up', but it looks like it's stuck, nothing is working. I get a 
> similar error for all hosts: “MgmtId 345050133919: Req: Resource [Host:11] is 
> unreachable:
> Host 11: Link is closed,” even if the hosts are reachable from 
> CloudStack and vSphere, where they are connected and manageable. I do 
> not see any events in vSphere.
>
>
>
> Any suggestions as to why CloudStack does not see that the servers are 
> reachable or why I get a timeout?
>
>
>
> Thank you,
>
> Cristian
>

 

Re: Issues migrating primary storage

[Jeremy Hansen]

I changed copy.volume.wait to 72000

But I just noticed:

kvm.storage.online.migration.wait and kvm.storage.offline.migration.wait. Worth 
changing this?

Thanks
-jeremy

> On Tuesday, Jan 16, 2024 at 11:01 PM, Jithin Raju  (mailto:jithin.r...@shapeblue.com)> wrote:
> Hi Jeremy,
>
> Have you checked the ‘wait’ parameter? Used as wait * 2 timeout.
>
> -Jithin
>
> From: Jeremy Hansen 
> Date: Wednesday, 17 January 2024 at 12:14 PM
> To: users@cloudstack.apache.org 
> Subject: Re: Issues migrating primary storage
> Unfortunately the upgrade didn’t help:
>
> Resource [StoragePool:3] is unreachable: Volume 
> [{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}] migration 
> failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to 
> copy 
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca
>  to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2].
>
>
>
> Anything else I can try? I’m trying to move away from NFS completely.
>
> -jeremy
>
>
>
>
>
>
> On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti 
> mailto:sureshkumar.anapa...@gmail.com>> wrote:
> Hi Jeremy,
>
> Can you extend with the config 'migratewait' and check.
>
> Regards,
> Suresh
>
> On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen 
> wrote:
>
>
> I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB
> volumes. These inevitably times out. I extended these configs:
>
> copy.volume.wait=72000
> job.cancel.threshold.minutes=480
> job.expire.minutes=1440
>
> This helped with smaller volumes but large once still eventually fail.
>
> 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196:
> Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> to
> b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}]
> }
>
> 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl]
> (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b)
> (logid:d7d98b81) Failed to migrate volume
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3]
> is unreachable: Volume
> [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}]
> migration failed due to [com.cloud.utils.exception.CloudRuntimeException:
> Failed to copy
> /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e
> to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2].
> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348)
> at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721)
> at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735)
> at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:16