Urgent : Network stuck in implementing state

[Bjoern Teipel]

Hi guys,

I had trouble while I was creating a new VLAN and now I can't delete it
anymore, because it's stuck in implementing state.
I guess that happened after the VR did not came up and I restarted the
management server so I could delete the VR and all addresses.

If I would spin up a guest, the VR and the guest would come up but I sill
want to get rid of it since I have an error in the network service offering.
After I cleaned all up though the GUI I can only see an external IP for the
old VR is still associated with the VLAN but can't be deleted (I don't get
it offered)

In the logs I found this and the NPE matches the table content :



2014-01-26 13:53:02,857 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Network id=212 is destroyed successfully, cleaning up corresponding
resources
 now.
2014-01-26 13:53:02,871 DEBUG [network.guru.DirectNetworkGuru]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Releasing ip 10.16.48.1 of placeholder nic Nic[189-null-null-10.16.48.1]
2014-01-26 13:53:02,872 DEBUG [db.Transaction.Transaction]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Rolling back the transaction: Time = 14 Name =
 -AsyncJobManagerImpl$1.run:494-Exec
utors$RunnableAdapter.call:471-FutureTask.run:262-ThreadPoolExecutor.runWorker:1145-ThreadPoolExecutor$Worker.run:615-Thread.run:744;
called by
-Transaction.rollback:897-Transaction.removeUpTo:840-Transaction.cl
ose:664-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroyNetwork:3144-ComponentInstantiationPostProcessor$Intercep
torDispatcher.intercept:125-NetworkServiceImpl.deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-AsyncJobManagerImp
l$1.run:531
2014-01-26 13:53:02,877 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Unexpected exception while executing
org.apache.cloudstack.api.command.user.ne
twork.DeleteNetworkCmd
java.lang.NullPointerException
at
com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:311)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:3144)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1767)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(DeleteNetworkCmd.java:70)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-26 13:53:02,880 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Complete async job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ],
jobStatus:
 2, resultCode: 530, result: Error Code: 530 Error text: null
2014-01-26 13:53:02,888 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Sync
queue (6) is currently empty
2014-01-26 13:53:02,889 WARN  [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Unable
to unregister active job [ 1127 ] = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24
020 ] from JMX monitoring


mysql> select * from nics  where id = 189\G
*** 1. row ***
id: 189
  uuid: 623d0af3-7ca5-4700-89a2-26252abdc054
   instance_id: NULL
   mac_address: NULL
   ip4_address: 10.16.48.1
   netmask: NULL
   gateway: NULL
   ip_type: NULL
 broadcast_uri: NULL
network_id: 212
  mode: NULL
 state: Reserved
  strategy: PlaceHolder
 reserver_name: NULL
reservation_id: NULL
 device_id: 0
   update_time: 2014-01-24 16:22:56
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 0
   vm_type: DomainRouter
   created: 2014-01-25 00:22:56
   removed: NULL
   ip6_gateway: NULL
  ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
1

RE: Cloudstack 4.2 on XenServer vs KVM

[John Mancuso]

All good advice- thanks everyone!

__
John Mancuso | System Engineer | FREEWHEEL
mobile: +1 516.652.2475  skype: jmancuso_freewheel
www.freewheel.tv 

SILICON VALLEY • NEW YORK • BEIJING  • LONDON


-Original Message-
From: Nux! [mailto:n...@li.nux.ro] 
Sent: Sunday, January 26, 2014 6:59 AM
To: users@cloudstack.apache.org
Subject: Re: Cloudstack 4.2 on XenServer vs KVM

On 26.01.2014 09:47, Geoff Higginbottom wrote:
> 
> As Shanker says, deploying a cloud with multiple hypervisors is a 
> common approach, ensuring you get the best of both worlds.

I think that's the bottom line, there's no one hv to rule them all, so test and 
use whatever fits your needs.

Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

Re: installing libvirt 1.0.0 or higher on CentOS 6.5

[Nux!]

On 26.01.2014 17:45, chris snow wrote:
I need to install the following versions of libvirt and qemu on CentOS 
6.5

as listed in the Apache Cloudstack 4.2.0 Installation Guide[1]:

 - libvirt: 1.0.0 or higher
 - qemu: 1.0 or higher

The CentOS versions are:

 - libvirt: 0.10.2
 - qemu: 0.12.1.2

Question 1) Where can I find rpms that meet the requirements, or do I 
need

to compile and install from source?

Question 2) If I need to install from source, are there some 
instructions

available?

  [1]:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/lxc-install.html


Not sure, but I think the stock CentOS 6.5 packages might be good 
enough.
Getting newer versions installed might be a bit tricky, not to mention 
you'd have to deal with security and updates yourself.
If you really must, you can try to rebuild newer Fedora SRPMS, but it's 
likely you will also need a newer kernel. Maybe using Ubuntu might be a 
better choice..


--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

installing libvirt 1.0.0 or higher on CentOS 6.5

[chris snow]

I need to install the following versions of libvirt and qemu on CentOS 6.5
as listed in the Apache Cloudstack 4.2.0 Installation Guide[1]:

 - libvirt: 1.0.0 or higher
 - qemu: 1.0 or higher

The CentOS versions are:

 - libvirt: 0.10.2
 - qemu: 0.12.1.2

Question 1) Where can I find rpms that meet the requirements, or do I need
to compile and install from source?

Question 2) If I need to install from source, are there some instructions
available?

  [1]:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/lxc-install.html

Re: Connect to a MySQL Galera Cluster

[Francois Gaudreault]

On 1/26/2014, 5:34 AM, Shanker Balan wrote:

The other way would be to make the whole fault tolerance an app problem. If
the app could automagically reconfigure itself to do round robin writes
across the galera cluster based on health checks, we will get “HA” 
automatically.
A failed node would end up being blacklisted by the app.
JDBC is supposed to be able to do that out of the box. But that means a 
code recompile. Something like:


jdbc:mysql:loadbalance://node1,node2,nodeN/database?loadBalanceBlacklistTimeout=5000

--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudrea...@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Connect to a MySQL Galera Cluster

[Francois Gaudreault]

I just noticed some DB HA parameters in the db.properties.in file. 
Anyone ever tried them?


It looks like CloudStack is able to handle slaves and autoReconnectForPool?

Francois

I was not able to get much information but Looks like it does not manage 
sessions
across the cluster. So if a node fails, all sessions currently handled by
the node will eventually timeout and fail.

It is up to the client now to open new connections.

One way to work around the session handling issue would be to use a
L7 proxy that can talk MySQL. I think mysql-proxy can do that but
its alpha(?) s/w. Also, mysql-proxy itself is not HA.

The other way would be to make the whole fault tolerance an app problem. If
the app could automagically reconfigure itself to do round robin writes
across the galera cluster based on health checks, we will get “HA” 
automatically.
A failed node would end up being blacklisted by the app.

As of now, I don’t see any other way other than to force a CloudStack restart
in response to a galera node failure.

Hoping someone with more db-foo can tell us more.

--
@shankerbalan

M: +91 98860 60539 | O: +91 (80) 67935867
shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
Bangalore - 560 055

Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure 
Support offers the 
best 24/7 SLA for CloudStack Environments.

Apache CloudStack Bootcamp training courses

**NEW!** CloudStack 4.2.1 training
18th-19th February 2014, Brazil. 
Classroom
17th-23rd March 2014, Region A. Instructor led, 
On-line
24th-28th March 2014, Region B. Instructor led, 
On-line
16th-20th June 2014, Region A. Instructor led, 
On-line
23rd-27th June 2014, Region B. Instructor led, 
On-line

This email and any attachments to it may be confidential and are intended solely 
for the use of the individual to whom it is addressed. Any views or opinions 
expressed are solely those of the author and do not necessarily represent those of 
Shape Blue Ltd or related companies. If you are not the intended recipient of this 
email, you must neither take any action based upon its contents, nor copy or show 
it to anyone. Please contact the sender if you believe you have received this email 
in error. Shape Blue Ltd is a company incorporated in England & Wales. 
ShapeBlue Services India LLP is a company incorporated in India and is operated 
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company 
incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue 
is a registered trademark.




--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudrea...@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Re: Cloudstack 4.2 on XenServer vs KVM

[Nux!]

On 26.01.2014 09:47, Geoff Higginbottom wrote:


As Shanker says, deploying a cloud with multiple hypervisors is a
common approach, ensuring you get the best of both worlds.


I think that's the bottom line, there's no one hv to rule them all, so 
test and use whatever fits your needs.


Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

Re: Connect to a MySQL Galera Cluster

[Nux!]

On 26.01.2014 04:40, Francois Gaudreault wrote:

Yes, we tried with a NetScaler, but as soon as one MySQL dies, both
management servers fence themselves on DB exceptions :S


I think part of your problem is you run only 2 galera nodes.
"Thus it is strongly advised that the minimum Galera cluster 
configuration is 3 nodes." - 
http://www.codership.com/wiki/doku.php?id=info


This is how we run it and it does indeed seem to be crucial for quorum.

Additionally we're using this software to deploy and manage it 
(partially), it can help with failover and recovery.

http://www.severalnines.com/clustercontrol

Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

Re: Connect to a MySQL Galera Cluster

[Shanker Balan]

Comments inline.

On 26-Jan-2014, at 11:14 am, Shanker Balan  wrote:

> Comments inline.
>
> On 26-Jan-2014, at 10:10 am, Francois Gaudreault  
> wrote:
>
>> Yes, we tried with a NetScaler, but as soon as one MySQL dies, both 
>> management servers fence themselves on DB exceptions :S
>
>
> Does Galera maintain client connection state across the failover?

I was not able to get much information but Looks like it does not manage 
sessions
across the cluster. So if a node fails, all sessions currently handled by
the node will eventually timeout and fail.

It is up to the client now to open new connections.

One way to work around the session handling issue would be to use a
L7 proxy that can talk MySQL. I think mysql-proxy can do that but
its alpha(?) s/w. Also, mysql-proxy itself is not HA.

The other way would be to make the whole fault tolerance an app problem. If
the app could automagically reconfigure itself to do round robin writes
across the galera cluster based on health checks, we will get “HA” 
automatically.
A failed node would end up being blacklisted by the app.

As of now, I don’t see any other way other than to force a CloudStack restart
in response to a galera node failure.

Hoping someone with more db-foo can tell us more.

--
@shankerbalan

M: +91 98860 60539 | O: +91 (80) 67935867
shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
Bangalore - 560 055

Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure 
Support offers the 
best 24/7 SLA for CloudStack Environments.

Apache CloudStack Bootcamp training courses

**NEW!** CloudStack 4.2.1 training
18th-19th February 2014, Brazil. 
Classroom
17th-23rd March 2014, Region A. Instructor led, 
On-line
24th-28th March 2014, Region B. Instructor led, 
On-line
16th-20th June 2014, Region A. Instructor led, 
On-line
23rd-27th June 2014, Region B. Instructor led, 
On-line

This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Ltd or related companies. If you are not the 
intended recipient of this email, you must neither take any action based upon 
its contents, nor copy or show it to anyone. Please contact the sender if you 
believe you have received this email in error. Shape Blue Ltd is a company 
incorporated in England & Wales. ShapeBlue Services India LLP is a company 
incorporated in India and is operated under license from Shape Blue Ltd. Shape 
Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: Cloudstack 4.2 on XenServer vs KVM

[Geoff Higginbottom]

Of all the cloud deployments we have been involved with (and that's a lot) 
XenServer has been the most popular Hypervisor by far.  A number of deployments 
were planning on using KVM but switched to XenServer after initial testing for 
one reason or another.

KVM is being used, and where it's a good fit it's a perfectly viable 
Hypervisor, especially when the organisation has KVM skills already in place.

As Shanker says, deploying a cloud with multiple hypervisors is a common 
approach, ensuring you get the best of both worlds.

Regards

Geoff Higginbottom
CTO / Cloud Architect


D: +44 20 3603 0542 | S: +44 20 3603 0540 
| M: +447968161581

geoff.higginbot...@shapeblue.com | 
www.shapeblue.com

ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 
4HS



On 26 Jan 2014, at 03:53, "Shanker Balan" 
mailto:shanker.ba...@shapeblue.com>> wrote:

Comments inline.

On 26-Jan-2014, at 8:35 am, Nux! mailto:n...@li.nux.ro>> wrote:

On 26.01.2014 00:39, John Mancuso wrote:
So, I am planning on setting up a brand new cloud infrastructure
using Cloudstack 4.2 on RHEL6. Cloudstack is hypervisor agnostic- I
got that... However there are some differences and features that are
available on XenServer that are not available on KVM. This is from a
Citrix salesperson:
"Here is some feedback on the following benefits of using Citrix
XenServer over KVM:
1.  Recurring Volume Snapshots with delta - Citrix XenServer is the
only hypervisor where recurring snapshots will be deltas (in other
hypervisors every volume snapshot is full) - this provides significant
space savings on secondary storage
2.  VM snapshots (taking a snapshot of a VM volumes including
memory state - not possible with KVM which supports only volume
snapshots)
3.  Live Storage Migration is only possible on Citrix XenServer
(not supported on KVM)
4.  Live CPU and Memory Scaling for running instances (not supported on KVM)"
On the Redhat side they have made it very clear that while Xen is
still available, KVM is the hypervisor technology they are pushing &
supporting going forward.
On the Apache/Citrix side, I get the feeling that from a QA
perspective CloudStack (and CloudPlatform) is based and tested on
XenServer and would be preferable in a stable & reliable  Production
environment.

Hello,

You are mostly correct, those points seem valid and right now Xenserver is the 
better supported hypervisor, it is quite mature and with loads of nice 
features. I'm seriously considering it myself.

Having said that, many clouds deployed nowadays are on KVM; yes it is missing 
some features but it has a huge user base, it's very stable and the performance 
is great; for me the killer feature is that I got a "real" OS as hypervisor, an 
OS that I have used extensively and am quite familiar with, for which we have 
deployment and monitoring infra in place etc etc. Additionally, if you want to 
use more exotic stuff, such as GlusterFS, Ceph or whatever crazy thing (CLVM 
over multiple mpath devices?) can run in RHEL/CentOS proper KVM is again the 
best choice. If you want VXLAN you are again limited to KVM afaik.

So it kind of depends on your needs, luckily there are good quality options to 
satisfy most of them.

What I personally find most relevant for choosing a hypervisor is guest OS
support. Not all hypervisors are created equal when it comes to supporting
a wide mix of operating systems.

Given that XenServer and KVM are both open source, I would recommend running
both. It gives you OS flexibility and spreads risks.

If you can afford, add VMware also. Enterprises love VMware.

Regards.

--
@shankerbalan

M: +91 98860 60539 | O: +91 (80) 67935867
shanker.ba...@shapeblue.com | 
www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
Bangalore - 560 055

Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure 
Support offers the 
best 24/7 SLA for CloudStack Environments.

Apache CloudStack Bootcamp training courses

**NEW!** CloudStack 4.2.1 training
18th-19th February 2014, Brazil. 
Classroom
17th-23rd March 2014, Region A. Instructor led, 
On-line
24th-28th March 2014, Region B. Instructor led, 
On-line
16th-20th June 2014, Region A. Instructor led, 
On-line
23rd-27th June 2014, Region B. Instructor led, 
On-line

This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Lt