Re: Recreating SystemVM's

[Makrand]

Hi there,

Looks more like hypervisor issue.

Just run *xe-toolstack-restart* on hosts where these VMs are trying to
start or if you don't have too many hosts, better run on all members
including master. most of i/o related issues squared off by toolstack
bounce.

--
Makrand


On Wed, Jun 7, 2017 at 3:01 AM, Jeremy Peterson 
wrote:

> Ok so I pulled this from Sunday morning.
>
> https://pastebin.com/nCETw1sC
>
>
> errorInfo: [HOST_CANNOT_ATTACH_NETWORK, 
> OpaqueRef:65d0c844-bd70-81e9-4518-8809e1dc0ee7,
> OpaqueRef:0093ac3f-9f3a-37e1-9cdb-581398d27ba2]
>
> XenServer error.
>
> Now this still gets me because all of the other VM's launched just fine.
>
> Going into XenCenter I see an error at the bottom This PIF is a bond slave
> and cannot be plugged.
>
> ???
>
> If I go to networking on the hosts I see the storage vlans and bonds are
> all there.
>
> I see my GUEST-PUB bond is there and LACP is setup correct.
>
> Any suggestions ?
>
>
> Jeremy
>
>
> -Original Message-
> From: Jeremy Peterson [mailto:jpeter...@acentek.net]
> Sent: Tuesday, June 6, 2017 9:23 AM
> To: users@cloudstack.apache.org
> Subject: RE: Recreating SystemVM's
>
> Thank you all for those responses.
>
> I'll comb through my management-server.log and post a pastebin if I'm
> scratching my head.
>
> Jeremy
>
> -Original Message-
> From: Rajani Karuturi [mailto:raj...@apache.org]
> Sent: Tuesday, June 6, 2017 6:53 AM
> To: users@cloudstack.apache.org
> Subject: Re: Recreating SystemVM's
>
> If the zone is enabled, cloudstack should recreate them automatically.
>
> ~ Rajani
>
> http://cloudplatform.accelerite.com/
>
> On June 6, 2017 at 11:37 AM, Erik Weber (terbol...@gmail.com)
> wrote:
>
> CloudStack should recreate automatically, check the mgmt server logs for
> hints of why it doesn't happen.
>
> --
> Erik
>
> tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson
> :
>
> I had an issue Sunday morning with cloudstack 4.9.0 and xenserver 6.5.0.
> My hosts stop sending LACP PDU's and caused a network drop to iSCSI
> primary storage.
>
> So all my instances recovered via HA enabled.
>
> But my console proxy and secondary storage system VM's got stuck in a boot
> state that would not power on.
>
> At this time they are expunged and gone.
>
> How do I tell cloudstack-management to recreate system VM's?
>
> I'm drawing a blank since deploying CS two years ago and just keeping
> things running and adding hosts and more storage everything has been so
> stable.
>
> Jeremy
>

RE: Recreating SystemVM's

[Jeremy Peterson]

Ok so I pulled this from Sunday morning.

https://pastebin.com/nCETw1sC


errorInfo: [HOST_CANNOT_ATTACH_NETWORK, 
OpaqueRef:65d0c844-bd70-81e9-4518-8809e1dc0ee7, 
OpaqueRef:0093ac3f-9f3a-37e1-9cdb-581398d27ba2]

XenServer error.

Now this still gets me because all of the other VM's launched just fine.

Going into XenCenter I see an error at the bottom This PIF is a bond slave and 
cannot be plugged.

???

If I go to networking on the hosts I see the storage vlans and bonds are all 
there.

I see my GUEST-PUB bond is there and LACP is setup correct.

Any suggestions ? 


Jeremy


-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net] 
Sent: Tuesday, June 6, 2017 9:23 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Thank you all for those responses.

I'll comb through my management-server.log and post a pastebin if I'm 
scratching my head.

Jeremy

-Original Message-
From: Rajani Karuturi [mailto:raj...@apache.org] 
Sent: Tuesday, June 6, 2017 6:53 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

If the zone is enabled, cloudstack should recreate them automatically.

~ Rajani

http://cloudplatform.accelerite.com/

On June 6, 2017 at 11:37 AM, Erik Weber (terbol...@gmail.com)
wrote:

CloudStack should recreate automatically, check the mgmt server logs for hints 
of why it doesn't happen.

--
Erik

tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson
:

I had an issue Sunday morning with cloudstack 4.9.0 and xenserver 6.5.0.
My hosts stop sending LACP PDU's and caused a network drop to iSCSI primary 
storage.

So all my instances recovered via HA enabled.

But my console proxy and secondary storage system VM's got stuck in a boot 
state that would not power on.

At this time they are expunged and gone.

How do I tell cloudstack-management to recreate system VM's?

I'm drawing a blank since deploying CS two years ago and just keeping things 
running and adding hosts and more storage everything has been so stable.

Jeremy

Retiring OLD Primary Storage - Dealing with System VMs and (assumed) Snapshot related artifacts

[David Merrill]

Hi All, 

We're in the process of phasing out primary storage in one of our zones and had 
some questions about dealing with: 


1. system VMs (Console Proxy & Secondary Storage) 
2. what looks like (to me), left over snapshot artifacts 

We're running CloudPlatform 4.5.1 with XenServer hypervisors and the storage to 
be retired are Dell Equallogic servers. 

For the first item I think all I need to do is mark the primary storage on the 
Dell EQL's into maintenance mode and then destroy them so that they'll come up 
on the new primary storage (already provisioned & configured in CloudPlatform). 

It's the second item that's giving me pause, when looking at the SR associated 
with the primary storage I'm seeing disks named like: 


* ABCServer_ROOT-685_20160905000250 

and in CloudPlatform have found an associated volume snapshot with the same 
date data. This seemed odd because I've understood that volume snapshots should 
be on secondary storage. 

I deleted the snapshot in CloudPlatform thinking that perhaps the disk above 
would disappear from view when looking at the SR in XenCenter, but it's still 
there. The impression I'm getting is that perhaps this is left over from some 
volume snapshot failure to clean up? 

How can I best chase this down? I'd like to retire this storage but I'd like to 
make sure that once it's gone there isn't something in the database that's 
still referring items there. 

Thanks, 
David 

 
A Message From... 
David Merrill - Senior Systems Engineer 


"Uptime. All the time." ® www.reliablenetworks.com 

Fifteen Years In Business 2003 - 2017 

477 Congress Street, Suite 812 | Portland, ME 04101 | ( 207) 772-5678 

private/hybrid cloud hosting | Zimbra groupware | managed services 
proactive maintenance and monitoring | technology consulting 

Maine's only managed services and cloud hosting provider with a 
SOC 2 Type II audit covering Security, Availability and Confidentiality 

This email may contain information that is privileged and confidential. 
If you suspect that you were not intended to receive it, please delete 
it and notify us as soon as possible. Thank you. 

RE: Recreating SystemVM's

[Jeremy Peterson]

Thank you all for those responses.

I'll comb through my management-server.log and post a pastebin if I'm 
scratching my head.

Jeremy

-Original Message-
From: Rajani Karuturi [mailto:raj...@apache.org] 
Sent: Tuesday, June 6, 2017 6:53 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

If the zone is enabled, cloudstack should recreate them automatically.

~ Rajani

http://cloudplatform.accelerite.com/

On June 6, 2017 at 11:37 AM, Erik Weber (terbol...@gmail.com)
wrote:

CloudStack should recreate automatically, check the mgmt server logs for hints 
of why it doesn't happen.

--
Erik

tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson
:

I had an issue Sunday morning with cloudstack 4.9.0 and xenserver 6.5.0.
My hosts stop sending LACP PDU's and caused a network drop to iSCSI primary 
storage.

So all my instances recovered via HA enabled.

But my console proxy and secondary storage system VM's got stuck in a boot 
state that would not power on.

At this time they are expunged and gone.

How do I tell cloudstack-management to recreate system VM's?

I'm drawing a blank since deploying CS two years ago and just keeping things 
running and adding hosts and more storage everything has been so stable.

Jeremy

Re: Recreating SystemVM's

[Rajani Karuturi]

If the zone is enabled, cloudstack should recreate them
automatically.

~ Rajani

http://cloudplatform.accelerite.com/

On June 6, 2017 at 11:37 AM, Erik Weber (terbol...@gmail.com)
wrote:

CloudStack should recreate automatically, check the mgmt server
logs for
hints of why it doesn't happen.

--
Erik

tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson
:

I had an issue Sunday morning with cloudstack 4.9.0 and xenserver
6.5.0.
My hosts stop sending LACP PDU's and caused a network drop to
iSCSI primary
storage.

So all my instances recovered via HA enabled.

But my console proxy and secondary storage system VM's got stuck
in a boot
state that would not power on.

At this time they are expunged and gone.

How do I tell cloudstack-management to recreate system VM's?

I'm drawing a blank since deploying CS two years ago and just
keeping
things running and adding hosts and more storage everything has
been so
stable.

Jeremy

RE: Recreating SystemVM's

[Paul Angus]

Hi Jeremy,

What state does CloudStack show your system VMs in ?


Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Erik Weber [mailto:terbol...@gmail.com] 
Sent: 06 June 2017 07:07
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

CloudStack should recreate automatically, check the mgmt server logs for hints 
of why it doesn't happen.

--
Erik

tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson :

> I had an issue Sunday morning with cloudstack 4.9.0 and xenserver 6.5.0.
> My hosts stop sending LACP PDU's and caused a network drop to iSCSI 
> primary storage.
>
> So all my instances recovered via HA enabled.
>
> But my console proxy and secondary storage system VM's got stuck in a 
> boot state that would not power on.
>
> At this time they are expunged and gone.
>
> How do I tell cloudstack-management to recreate system VM's?
>
> I'm drawing a blank since deploying CS two years ago and just keeping 
> things running and adding hosts and more storage everything has been 
> so stable.
>
> Jeremy
>

Re: Recreating SystemVM's

[Erik Weber]

CloudStack should recreate automatically, check the mgmt server logs for
hints of why it doesn't happen.

-- 
Erik

tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson :

> I had an issue Sunday morning with cloudstack 4.9.0 and xenserver 6.5.0.
> My hosts stop sending LACP PDU's and caused a network drop to iSCSI primary
> storage.
>
> So all my instances recovered via HA enabled.
>
> But my console proxy and secondary storage system VM's got stuck in a boot
> state that would not power on.
>
> At this time they are expunged and gone.
>
> How do I tell cloudstack-management to recreate system VM's?
>
> I'm drawing a blank since deploying CS two years ago and just keeping
> things running and adding hosts and more storage everything has been so
> stable.
>
> Jeremy
>