Hi
I am trying to investigate a problem I have with Apache CloudStack when
creating an instance using Apache
Brooklyn. I have installed Brooklyn 0.9 on a CentOS 6.8 minimal server and
Cloudstack 4.9 on two
CentOS 6.8 minimal servers. I have a simple piece of yaml code.
name: Cassandra 1
location:
jclouds:cloudstack:
endpoint: http://192.168.1.109:8080/client/api
identity: Jo4lB5oJqWVLJ1vSQjiN9O5UmkeqylGyyyyy
credential: CJqAGj9rUCAyrSAYWCrjXq2-_xEaRxxxxx
services:
- type: org.apache.brooklyn.entity.nosql.cassandra.CassandraNode
brookyn.policies:
- type: org.apache.brooklyn.policy.ha.ServiceRestarter
brooklyn.enrichers:
- type: org.apache.brooklyn.policy.ha.ServiceFailureDetector
I have a cloud zone up and running with a basic network type, everything
seems healthy. The Zone's physical
network has been created with guest and management Ip address ranges.
uest
Pod Gateway Netmask Start IP End IP
Pod1 192.168.1.1 255.255.255.0 192.168.1.151 192.168.1.170
Management
Pod Gateway Netmask Start IP End IP
Pod1 192.168.1.1 255.255.255.0 192.168.1.130 192.168.1.150
No user defined instances are running on the cloud, only the system defined
VM's.
mysql> mysql
mysql> use cloud ;
mysql> select
id,public_ip_address,allocated,state,is_system,removed,created
from user_ip_address where state <> 'Free';
+----+-------------------+---------------------+-----------+-----------+---------+---------+
| id | public_ip_address | allocated | state | is_system |
removed | created |
+----+-------------------+---------------------+-----------+-----------+---------+---------+
| 15 | 192.168.1.165 | 2016-09-05 02:41:08 | Allocated | 0 |
NULL | NULL | => Secondary Storage VM
| 16 | 192.168.1.166 | 2016-09-05 02:41:06 | Allocated | 0 |
NULL | NULL | => Console Proxy VM
| 17 | 192.168.1.167 | 2016-10-15 01:43:26 | Allocated | 0 |
NULL | NULL | => r-4-VM virtual router
+----+-------------------+---------------------+-----------+-----------+---------+---------+
In brooklyn I attempt to run a simple cassandra based blueprint to
dynamically create a single instance
on the cloud. By default the cloud has a template "CentOS 5.5(64-bit) no
GUI (KVM)" and that will be
used in instance creation.
So apart from the ip addresses used by the system i.e. 165 - 167, I think
all IP addresses in the ranges
above are free. However, when I execute a blue print I receive an error.
output from brooklyn terminal gives rest api call
1) ExecutionException on
brooklyn-offizb-root-cassand-gc5r-apache-cassandra-nz4u-ef0:
java.util.concurrent.ExecutionException:
org.jclouds.http.HttpResponseException: command: GET
http://192.168.1.109:8080/client/api?
response=json&command=associateIpAddress&zoneid=eccbe4cc-a518-4b2b-a8d3-c623b0460463&networkid=4ece6bff-6776-4f87-84b1-3f7eaac30850
HTTP/1.1 failed with response: HTTP/1.1 533 null; content:
[{"associateipaddressresponse":{"uuidList":
[],"errorcode":533,"cserrorcode":9999,"errortext":"Insufficient address
capacity"}}]
------------------------------------------------------------------------
and similar errors in the cloudstack management logs.
------------------------------------------------------------------------
[root@hc4nn management]# hostname ; pwd
hc4nn
/var/log/cloudstack/management
apilog.log:2016-10-16 11:29:13,920 INFO [a.c.c.a.ApiServer]
(catalina-exec-17:ctx-3774395e ctx-2a
ee0f83 ctx-4b529848) (logid:b125de78) (userId=2 accountId=2 sessionId=null)
192.168.1.118 -- GET r
esponse=json&command=associateIpAddress&zoneid=eccbe4cc-a518-4b2b-a8d3-c623b0460463&networkid=4ece
6bff-6776-4f87-84b1-3f7eaac30850&apiKey=Jo4lB5oJqWVLJ1vSQjiN9O5UmkeqylGI1_OVwNHdHdLUPhoHda2b8jGylR
cj8TeRVbKmbvPe3ZFnTwoAwYf1UQ&signature=jCOeBAEmpPlkSqgpVcp/hi6IyfA%3D 533
Insufficient address cap
acity
management-server.log:2016-10-16 11:29:13,818 INFO
[o.a.c.a.c.u.a.AssociateIPAddrCmd] (catalina-e
xec-7:ctx-ca17b8bb ctx-5c175b06 ctx-ad528fed) (logid:ad36d5dd)
com.cloud.exception.InsufficientAdd
ressCapacityException: Insufficient address capacityScope=interface
com.cloud.dc.DataCenter; id=1
management-server.log:2016-10-16 11:29:13,818 TRACE
[o.a.c.a.c.u.a.AssociateIPAddrCmd] (catalina-e
xec-7:ctx-ca17b8bb ctx-5c175b06 ctx-ad528fed) (logid:ad36d5dd)
com.cloud.exception.InsufficientAdd
ressCapacityException: Insufficient address capacityScope=interface
com.cloud.dc.DataCenter; id=1
management-server.log:2016-10-16 11:29:13,819 INFO [c.c.a.ApiServer]
(catalina-exec-7:ctx-ca17b8b
b ctx-5c175b06 ctx-ad528fed) (logid:ad36d5dd) Insufficient address capacity
It seems that I am short of IP addresses but the IP address assigned to the
new cloud stack instance
i.e. 160 should have been free. Also, in Cloudstack I can easily recover a
detroyed instance,
destroyed beause of this error and it will run.
I would like to get some pointers as to how to investigate this error as Im
a little stumped. Thanks
in advance. Is this an issue with Brooklyn or Cloudstack or do I need to
configure more resources
in some way. Im going to mail to both the Brooklyn and Cloudstack mailing
lists to try and understand
the problem from both directions.
cheers
Mike
