firewall problems and proxy problems
I've started out with cloudstack a couple days ago and I've hit a bit of a
brick wall.
The installation is running on a system that has two network interfaces, a
public one and private one (connecting to a local network).
auto eth0
iface eth0 inet manual
auto eth1
iface eth1 inet static
address 10.158.231.29
netmask 255.255.255.0
network 10.158.231.0
broadcast 10.158.231.255
gateway 10.158.231.1
auto cloudbr0
iface cloudbr0 inet static
address 172.16.8.7
netmask 255.255.255.0
network 172.16.8.0
broadcast 172.16.8.255
up route add -net 172.16.0.0 netmask 255.255.0.0 gw 172.16.8.1
bridge_ports eth0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
The idea is to connect all the guests to the 172.16 network because access
to multicasting devices is needed.
The systems are running on Debian wheezy, and got the system up after
fixing the /etc/legal problem (echo Ubuntu) and after finding out in the
logs that there is a package dependency problem on the cloudstack-manager
package for 4.3
The biggest problem seems to be that host is running in a corporate network
and cloudstack is auto configuring some things by downloading over http.
This does not work since a corporate proxy is required.
I downloaded the system images manually; but now it seems I need to do the
same in the secondary storage vm.
The vm is running and I can get the login prompt with virsh (kvm), however
there is no password I can find.
The trick with the connection to the local-link address does not work
because I fear I have a networking/firewall issue. It does not matter if I
disable the firewall or not, I cannot access the 169.254.x.x network.
pinging the device returns
ping 169.254.3.236
PING 169.254.3.236 (169.254.3.236) 56(84) bytes of data.
>From 169.254.0.1 icmp_seq=1 Destination Host Unreachable
>From 169.254.0.1 icmp_seq=2 Destination Host Unreachable
>From 169.254.0.1 icmp_seq=3 Destination Host Unreachable
The firewall configuration is firehol
cat /etc/firehol/firehol.conf
version 5
# Accept all client traffic on any interface
FIREHOL_LOG_MODE="ULOG"
server_cloudstackweb_ports="tcp/8080"
client_cloudstackweb_ports="default"
server_buildbot_ports="tcp/8010"
client_buildbot_ports="default"
server_git_ports="tcp/9418"
client_git_ports="default"
labo_ips="172.16.0.0/16 169.254.0.0/16"
server_cloudstack_ports="tcp/1798"
client_cloudstack_ports="default"
server_libvirt_ports="tcp/16509"
client_libvirt_ports="default"
server_vnc_ports="tcp/5900:6100"
client_vnc_ports="default"
server_libvirtlive_ports="tcp/49152:49216"
client_libvirtlive_ports="default"
# local bridge address
interface cloudbr0 LAN src "${labo_ips}"
server all accept
client all accept
interface eth1 WAN src not "${labo_ips}"
protection strong
policy reject
server ssh accept
server cloudstack accept
server cloudstackweb accept
server libvirt accept
server vnc accept
server libvirtlive accept
server buildbot accept
client all accept
# zeroconf bridge address
interface cloud0 LLBR0
client all accept
server all accept
router LAN2WAN inface cloudbr0 outface eth1
masquerade
route all accept
I got a bit further by adding the interface cloud0 to the definition, but
still no joy.
I need to access the vm to run some script with a http_proxy environment
variable...
I am currently starting to run out of ideas; installing cloudstack was
fine; but I entered hell afterwards; ...
/sbin/ifconfig
cloud0Link encap:Ethernet HWaddr fe:00:a9:fe:01:f8
inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::4061:7aff:fe05:f240/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:4278 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:193752 (189.2 KiB)
cloudbr0 Link encap:Ethernet HWaddr 9c:8e:99:26:6d:e4
inet addr:172.16.8.7 Bcast:172.16.8.255 Mask:255.255.255.0
inet6 addr: fe80::9e8e:99ff:fe26:6de4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:123565 errors:0 dropped:0 overruns:0 frame:0
TX packets:58009 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:42680449 (40.7 MiB) TX bytes:236280976 (225.3 MiB)
eth0 Link encap:Ethernet HWaddr 9c:8e:99:26:6d:e4
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:930187 errors:0 dropped:871 overruns:0 frame:0
TX packets:699239 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:314733647 (300.1 MiB) TX bytes:565773302 (539.5 MiB)
eth1 Link encap:Ethernet HWaddr 9c:8e:99:26:6d:e6
inet addr:10.158.231.29 Bcast:150
Re: firewall problems and proxy problems
I've managed to get link local working my modifying the routing (disable the fw atm too). /sbin/route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 0.0.0.0 10.158.231.1 0.0.0.0 UG0 00 eth1 150.158.231.0 0.0.0.0 255.255.255.0 U 0 00 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 00 cloudbr0 172.16.0.0 0.0.0.0 255.255.0.0 U 0 00 cloudbr0 172.16.8.0 0.0.0.0 255.255.255.0 U 0 00 cloudbr0 However, only to OTHER machines, I still cannot access the Secondary storage VM by the ll address. So LL addresses work to other physical machines, not to the VM on the same host I am connecting from (or from another machine for that matter).
Fwd: firewall problems and proxy problems
I think my mail did not get through, I can't find it in the archives; I've
probably sent it too fast after subscribing
I've started out with cloudstack a couple days ago and I've hit a bit of a
brick wall.
The installation is running on a system that has two network interfaces, a
public one and private one (connecting to a local network).
auto eth0
iface eth0 inet manual
auto eth1
iface eth1 inet static
address 10.158.231.29
netmask 255.255.255.0
network 10.158.231.0
broadcast 10.158.231.255
gateway 10.158.231.1
auto cloudbr0
iface cloudbr0 inet static
address 172.16.8.7
netmask 255.255.255.0
network 172.16.8.0
broadcast 172.16.8.255
up route add -net 172.16.0.0 netmask 255.255.0.0 gw 172.16.8.1
bridge_ports eth0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
The idea is to connect all the guests to the 172.16 network because access
to multicasting devices is needed.
The systems are running on Debian wheezy, and got the system up after
fixing the /etc/legal problem (echo Ubuntu) and after finding out in the
logs that there is a package dependency problem on the cloudstack-manager
package for 4.3
The biggest problem seems to be that host is running in a corporate network
and cloudstack is auto configuring some things by downloading over http.
This does not work since a corporate proxy is required.
I downloaded the system images manually; but now it seems I need to do the
same in the secondary storage vm.
The vm is running and I can get the login prompt with virsh (kvm), however
there is no password I can find.
The trick with the connection to the local-link address does not work
because I fear I have a networking/firewall issue. It does not matter if I
disable the firewall or not, I cannot access the 169.254.x.x network.
pinging the device returns
ping 169.254.3.236
PING 169.254.3.236 (169.254.3.236) 56(84) bytes of data.
>From 169.254.0.1 icmp_seq=1 Destination Host Unreachable
>From 169.254.0.1 icmp_seq=2 Destination Host Unreachable
>From 169.254.0.1 icmp_seq=3 Destination Host Unreachable
The firewall configuration is firehol
cat /etc/firehol/firehol.conf
version 5
# Accept all client traffic on any interface
FIREHOL_LOG_MODE="ULOG"
server_cloudstackweb_ports="tcp/8080"
client_cloudstackweb_ports="default"
server_buildbot_ports="tcp/8010"
client_buildbot_ports="default"
server_git_ports="tcp/9418"
client_git_ports="default"
labo_ips="172.16.0.0/16 169.254.0.0/16"
server_cloudstack_ports="tcp/1798"
client_cloudstack_ports="default"
server_libvirt_ports="tcp/16509"
client_libvirt_ports="default"
server_vnc_ports="tcp/5900:6100"
client_vnc_ports="default"
server_libvirtlive_ports="tcp/49152:49216"
client_libvirtlive_ports="default"
# local bridge address
interface cloudbr0 LAN src "${labo_ips}"
server all accept
client all accept
interface eth1 WAN src not "${labo_ips}"
protection strong
policy reject
server ssh accept
server cloudstack accept
server cloudstackweb accept
server libvirt accept
server vnc accept
server libvirtlive accept
server buildbot accept
client all accept
# zeroconf bridge address
interface cloud0 LLBR0
client all accept
server all accept
router LAN2WAN inface cloudbr0 outface eth1
masquerade
route all accept
I got a bit further by adding the interface cloud0 to the definition, but
still no joy.
I need to access the vm to run some script with a http_proxy environment
variable...
I am currently starting to run out of ideas; installing cloudstack was
fine; but I entered hell afterwards; ...
/sbin/ifconfig
cloud0Link encap:Ethernet HWaddr fe:00:a9:fe:01:f8
inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::4061:7aff:fe05:f240/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:4278 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:193752 (189.2 KiB)
cloudbr0 Link encap:Ethernet HWaddr 9c:8e:99:26:6d:e4
inet addr:172.16.8.7 Bcast:172.16.8.255 Mask:255.255.255.0
inet6 addr: fe80::9e8e:99ff:fe26:6de4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:123565 errors:0 dropped:0 overruns:0 frame:0
TX packets:58009 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:42680449 (40.7 MiB) TX bytes:236280976 (225.3 MiB)
eth0 Link encap:Ethernet HWaddr 9c:8e:99:26:6d:e4
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:930187 errors:0 dropped:871 overruns:0 frame:0
TX packets:699239 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:314733647 (300.1 MiB) TX bytes:565
Re: firewall problems and proxy problems
OK, there is something off here: This is my routing table where I can ping another physical host with an link-local address route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 0.0.0.0 10.158.231.1 0.0.0.0 UG0 00 eth1 10.158.231.0 0.0.0.0 255.255.255.0 U 0 00 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 00 cloudbr0 172.16.0.0 0.0.0.0 255.255.0.0 U 0 00 cloudbr0 172.16.8.0 0.0.0.0 255.255.255.0 U 0 00 cloudbr0 If I restart the cloudstack-agent; the routing table has changed to: route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 0.0.0.0 10.158.231.1 0.0.0.0 UG0 00 eth1 10.158.231.0 0.0.0.0 255.255.255.0 U 0 00 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 00 cloud0 172.16.0.0 0.0.0.0 255.255.0.0 U 0 00 cloudbr0 172.16.8.0 0.0.0.0 255.255.255.0 U 0 00 cloudbr0 and link-local devices cannot be accessed. In both cases, I cannot access the system VMs with ssh.
Re: firewall problems and proxy problems
Marc, The only difference I see is de vice cloudbr0 becoming cloud0, in the third line. Am I correct? On Thu, Oct 16, 2014 at 3:28 PM, Marc Leeman wrote: > > OK, there is something off here: > > This is my routing table where I can ping another physical host with an > link-local address > > route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric RefUse > Iface > 0.0.0.0 10.158.231.1 0.0.0.0 UG0 00 eth1 > 10.158.231.0 0.0.0.0 255.255.255.0 U 0 00 eth1 > 169.254.0.0 0.0.0.0 255.255.0.0 U 0 00 > cloudbr0 > 172.16.0.0 0.0.0.0 255.255.0.0 U 0 00 > cloudbr0 > 172.16.8.0 0.0.0.0 255.255.255.0 U 0 00 > cloudbr0 > > > If I restart the cloudstack-agent; the routing table has changed to: > > route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric RefUse > Iface > 0.0.0.0 10.158.231.1 0.0.0.0 UG0 00 eth1 > 10.158.231.0 0.0.0.0 255.255.255.0 U 0 00 eth1 > 169.254.0.0 0.0.0.0 255.255.0.0 U 0 00 > cloud0 > 172.16.0.0 0.0.0.0 255.255.0.0 U 0 00 > cloudbr0 > 172.16.8.0 0.0.0.0 255.255.255.0 U 0 00 > cloudbr0 > > and link-local devices cannot be accessed. > > In both cases, I cannot access the system VMs with ssh. > -- Daan
Re: firewall problems and proxy problems
yes. But there must be something in the configuration that causes problems. I haven't played around too much with bridging. If I delete the route to cloud0 and replace it to the cloudbr0, I can access the LL devices on the network; after resetting the cloudstack manager; I cannot. It seems to my (untrained and unexpertised eye) that there is some link or something missing between cloud0 and cloudbr0
Re: firewall problems and proxy problems
My eye is not much more trained at this table but I'll have a peek in the code. It seems to e that cloudstack should be configuring the bridge instead of the cloud0 device. I have no idea if this is a config err of yours or a bug. My assuptions: you are using kvm as hypervisor you are running version 4.3.1 any other info? I might want to see your logs during install and zone definition later, you still have them? On Thu, Oct 16, 2014 at 3:55 PM, Marc Leeman wrote: > yes. > > But there must be something in the configuration that causes problems. I > haven't played around too much with bridging. > > If I delete the route to cloud0 and replace it to the cloudbr0, I can > access the LL devices on the network; after resetting the cloudstack > manager; I cannot. It seems to my (untrained and unexpertised eye) that > there is some link or something missing between cloud0 and cloudbr0 > -- Daan
Re: firewall problems and proxy problems
On 16 October 2014 16:10, Daan Hoogland wrote: > >> My eye is not much more trained at this table but I'll have a peek in the >> code. It seems to e that cloudstack should be configuring the bridge >> instead of the cloud0 device. I have no idea if this is a config err of >> yours or a bug. >> >> My assuptions: >> you are using kvm as hypervisor >> > > ack. > > >> you are running version 4.3.1 >> > > dpkg -l |grep cloudstack > ii cloudstack-agent 4.3.1 > all CloudStack agent > ii cloudstack-cli 4.3.1 > all The CloudStack CLI called CloudMonkey > ii cloudstack-common4.3.1 > all A common package which contains files which are shared by > several CloudStack packages > ii cloudstack-management4.3.1 > all CloudStack server library > > > >> >> any other info? I might want to see your logs during install and zone >> definition later, you still have them? >> > > > I still have all the log files on the server. > > The VMs should be connected to the same 172.16.x.x network as the host > because of access to multicast devices is needed. > > For the rest, we've been looking around on the system until it started > dawning on us that there were some scripts trying to download things and > got blocked by not using the proxy. > > We could not figure out whay the SVMs said starting; but got stuck there. > > I downloaded the SVM images with the script and the VMS started. > The log file is http://crichton.supercomputerrobot.com/~marc/downloads/management-server.log.2014-10-14.gz and the firewall has been disabled already; so it should not be dependent on this.
Re: firewall problems and proxy problems
Marc from your logs it seems like there is a port conflict on your host. Can you check on that? port 9090 first I see 2014-10-14 17:25:54,181 INFO [c.c.c.ClusterManagerImpl] (main:null) Management server (host id : 1) is being started at 172.16.8.7:9090 2014-10-14 17:25:54,186 INFO [c.c.c.ClusterManagerImpl] (main:null) Cluster manager was started successfully 2014-10-14 17:25:54,205 ERROR [c.c.u.n.NioConnection] (AgentManager-Selector:null) Unable to initialize the threads. java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70) at com.cloud.utils.nio.NioServer.init(NioServer.java:50) at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108) at java.lang.Thread.run(Thread.java:636) and then 2014-10-14 18:18:46,695 ERROR [c.c.c.ClusterManagerImpl] (main:null) Unable to ping management server at 172.16.8.7:9090 due to ConnectException are you installing a clustered set of management servers on a single host? (still looking) On Thu, Oct 16, 2014 at 4:33 PM, Marc Leeman wrote: > On 16 October 2014 16:10, Daan Hoogland wrote: > > > >> My eye is not much more trained at this table but I'll have a peek in > the > >> code. It seems to e that cloudstack should be configuring the bridge > >> instead of the cloud0 device. I have no idea if this is a config err of > >> yours or a bug. > >> > >> My assuptions: > >> you are using kvm as hypervisor > >> > > > > ack. > > > > > >> you are running version 4.3.1 > >> > > > > dpkg -l |grep cloudstack > > ii cloudstack-agent 4.3.1 > > all CloudStack agent > > ii cloudstack-cli 4.3.1 > > all The CloudStack CLI called CloudMonkey > > ii cloudstack-common4.3.1 > > all A common package which contains files which are shared by > > several CloudStack packages > > ii cloudstack-management4.3.1 > > all CloudStack server library > > > > > > > >> > >> any other info? I might want to see your logs during install and zone > >> definition later, you still have them? > >> > > > > > > I still have all the log files on the server. > > > > The VMs should be connected to the same 172.16.x.x network as the host > > because of access to multicast devices is needed. > > > > For the rest, we've been looking around on the system until it started > > dawning on us that there were some scripts trying to download things and > > got blocked by not using the proxy. > > > > We could not figure out whay the SVMs said starting; but got stuck there. > > > > I downloaded the SVM images with the script and the VMS started. > > > > The log file is > > > http://crichton.supercomputerrobot.com/~marc/downloads/management-server.log.2014-10-14.gz > > and the firewall has been disabled already; so it should not be dependent > on this. > -- Daan
Re: firewall problems and proxy problems
$ sudo netstat -tulpn |grep 9090 tcp6 0 0 :::9090 :::* LISTEN 32427/jsvc.exec That is cloudstack. But then I thought of it; I asked someone on the cloudstack stand at Dusseldorf; and the 4.0 was upgraded to 4.3. It seems that there was something wrong with the packages there: the old 4.0 manager/agents were not properly shutdown before they were replaced by the cloudstack version of the packages (the previous were called cloud). It might be this that you are noticing. I killed the processes manually and restarted the cloudstack application.
Re: firewall problems and proxy problems
Ok, let us know if this solves your issue. Op 16 okt. 2014 17:07 schreef "Marc Leeman" : > $ sudo netstat -tulpn |grep 9090 > tcp6 0 0 :::9090 :::* > LISTEN 32427/jsvc.exec > > That is cloudstack. > > But then I thought of it; I asked someone on the cloudstack stand at > Dusseldorf; and the 4.0 was upgraded to 4.3. It seems that there was > something wrong with the packages there: the old 4.0 manager/agents were > not properly shutdown before they were replaced by the cloudstack version > of the packages (the previous were called cloud). > > It might be this that you are noticing. > > I killed the processes manually and restarted the cloudstack application. > >
Re: firewall problems and proxy problems
No: I killed the process at the boot the day before yesterday (thanks to the cloudstack guys that I made to miss the boot crawl): there was indeed a problem with that; NO vsm were starting. After killing the old stuck programs; we got further and after downloading the svm images manually with the http_proxy var; the svms started. However then we hit the bridging problem.
Re: firewall problems and proxy problems
The port is being started by the manager. The error is happening during the restart of the management server 2014-10-16 17:27:36,909 ERROR [c.c.c.ClusterManagerImpl] (main:null) Unable to ping management server at 172.16.8.7:9090 due to ConnectException java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525) at com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1122) at com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1091) at com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1169) at com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1058) . However, connecting afterwards with telnet is fine: telnet 172.16.8.7 9090 Trying 172.16.8.7... Connected to 172.16.8.7. Escape character is '^]'. ^] root@zee:/var/log/cloudstack/management# ps -ef |grep 32427 Non-standard uts for running kernel: release Linux version 3.14-0.bpo.1-amd64 (debian-ker...@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.14.4-1~bpo70+1 (2014-05-14) =3.14.0 gives version code 200192 root 4553 4514 0 17:26 pts/600:00:00 grep 32427 cloud32427 32424 1 15:26 ?00:01:43 jsvc.exec -user cloud -cp /usr/share/java/commons-daemon.jar:/usr/share/cloudstack-management/bin/bootstrap.jar:/etc/cloudstack/management:/usr/share/cloudstack-management/setup -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/cloudstack-management.pid -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M -XX:MaxPermSize=800m -Djava.endorsed.dirs=/usr/share/cloudstack-management/endorsed -Dcatalina.base=/usr/share/cloudstack-management -Dcatalina.home=/usr/share/cloudstack-management -Djava.io.tmpdir=/tmp/cloudstack-management-temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties org.apache.catalina.startup.Bootstrap root@zee:/var/log/cloudstack/management# /etc/init.d/cloudstack-management restart [ ok ] Stopping CloudStack-specific Tomcat servlet engine: cloudstack-management. [ ok ] Starting CloudStack-specific Tomcat servlet engine: cloudstack-management. root@zee:/var/log/cloudstack/management# netstat -tulpn |grep 9090 root@zee:/var/log/cloudstack/management# netstat -tulpn |grep 9090 root@zee:/var/log/cloudstack/management# netstat -tulpn |grep 9090 root@zee:/var/log/cloudstack/management# netstat -tulpn |grep 9090 root@zee:/var/log/cloudstack/management# netstat -tulpn |grep 9090 tcp6 0 0 :::9090 :::* LISTEN 4643/jsvc.exec root@zee:/var/log/cloudstack/management# ps -ef |grep 4643 Non-standard uts for running kernel: release Linux version 3.14-0.bpo.1-amd64 (debian-ker...@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.14.4-1~bpo70+1 (2014-05-14) =3.14.0 gives version code 200192 cloud 4643 4640 99 17:27 ?00:00:52 jsvc.exec -user cloud -cp /usr/share/java/commons-daemon.jar:/usr/share/cloudstack-management/bin/bootstrap.jar:/etc/cloudstack/management:/usr/share/cloudstack-management/setup -outfile SYSLOG -errfile SYSLOG -pidfile /var/run/cloudstack-management.pid -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M -XX:MaxPermSize=800m -Djava.endorsed.dirs=/usr/share/cloudstack-management/endorsed -Dcatalina.base=/usr/share/cloudstack-management -Dcatalina.home=/usr/share/cloudstack-management -Djava.io.tmpdir=/tmp/cloudstack-management-temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties org.apache.catalina.startup.Bootstrap root 4999 4514 0 17:27 pts/600:00:00 grep 4643 On 16 October 2014 17:16, Marc Leeman wrote: > > No: > > I killed the process at the boot the day before yesterday (thanks to the > cloudstack guys that I made to miss the boot crawl): there was indeed a > problem with that; NO vsm were starting. > > After killing the old stuck programs; we got further and after downloading > the svm images manually with the http_proxy var; the svms started. > > However then we hit the bridging problem. > > > >
Re: firewall problems and proxy problems
I don't see a mention of cloud0 in the log. It might come from within the agent itself trying to restore some defaults after a reset or delete. I do note some other perculiar thing: Around the following log fragment a lot of nulls are in the file. Also the meassage is not filling one with hope: note that the nuls are output from 'lsmod|grep kvm'. Is this the problem you tackeled in Dusseldorf? 2014-10-14 17:59:05,664 DEBUG [c.c.u.s.SSHCmdHelper] (catalina-exec-11:ctx-ba00ae78 ctx-b1389dca) Executing cmd: cloudstack-setup-agent -m 10.158.231.29 -z 1 -p 1 -c 1 -g 4a9064dc-4e61-3c36-949f-d474d64f5e56 -a --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 2014-10-14 17:59:06,774 DEBUG [c.c.u.s.SSHCmdHelper] (catalina-exec-11:ctx-ba00ae78 ctx-b1389dca) cloudstack-setup-agent -m 10.158.231.29 -z 1 -p 1 -c 1 -g 4a9064dc-4e61-3c36-949f-d474d64f5e56 -a --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 output:Starting to configure your system: Checking KVM...[Failed] Please enable KVM on this machine Try to restore your system: On Thu, Oct 16, 2014 at 5:16 PM, Marc Leeman wrote: > No: > > I killed the process at the boot the day before yesterday (thanks to the > cloudstack guys that I made to miss the boot crawl): there was indeed a > problem with that; NO vsm were starting. > > After killing the old stuck programs; we got further and after downloading > the svm images manually with the http_proxy var; the svms started. > > However then we hit the bridging problem. > -- Daan
Re: firewall problems and proxy problems
> configure your system: > Checking KVM...[Failed] > Please enable KVM on this machine I opened a bug on this one: there is a packaging problem: a dependency is missing on cpu_utils. We got that one figured out too :-) The script is using kvm-ok at some point.
Re: firewall problems and proxy problems
Is there some inconsistency in the database? At the end of the stack dump: 2014-10-16 17:27:36,912 INFO [c.c.c.ClusterManagerImpl] (main:null) Detected that another management node with the same IP 172.16.8.7 is considered as running in DB, however it is not pingable, we will continue cluster initialization with this management server node
Re: firewall problems and proxy problems
> Try to restore your system: > You are talking about this procedure right? https://support.getcloudservices.com/entries/21982811-CloudStack-Restore-Cloud-Server
Re: firewall problems and proxy problems
no, this is restoring an instance AFAICT On Thu, Oct 16, 2014 at 5:43 PM, Marc Leeman wrote: > > Try to restore your system: > > > > > > You are talking about this procedure right? > > > https://support.getcloudservices.com/entries/21982811-CloudStack-Restore-Cloud-Server > -- Daan
Re: firewall problems and proxy problems
btw On Thu, Oct 16, 2014 at 5:46 PM, Daan Hoogland wrote: > no, this is restoring an instance AFAICT > > On Thu, Oct 16, 2014 at 5:43 PM, Marc Leeman > wrote: > >> > Try to restore your system: > > this was part of the log! Not an advice -- Daan
Re: firewall problems and proxy problems
On Thu, Oct 16, 2014 at 5:33 PM, Marc Leeman wrote: > Is there some inconsistency in the database? > This might well be. To check this look at mshost and mshost_peer. Are you upgrading? Is this a test setup? Daan
RE: firewall problems and proxy problems
Hi Marc,
I'm not sure if this is adding to your issues, but the section of your network
config where you send traffic for 172.16.0.0/16 to 172.16.8.1
doesn't look right to my eye, as traffic for your local subnet 172.16.8.0/24
would also get sent to that gateway.
address 172.16.8.7
netmask 255.255.255.0
network 172.16.8.0
broadcast 172.16.8.255
up route add -net 172.16.0.0 netmask 255.255.0.0 gw 172.16.8.1
Regards
Paul Angus
Cloud Architect
S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
paul.an...@shapeblue.com
-Original Message-
From: Marc Leeman [mailto:marc.lee...@gmail.com]
Sent: 16 October 2014 08:25
To: users@cloudstack.apache.org
Subject: Fwd: firewall problems and proxy problems
I think my mail did not get through, I can't find it in the archives; I've
probably sent it too fast after subscribing
I've started out with cloudstack a couple days ago and I've hit a bit of a
brick wall.
The installation is running on a system that has two network interfaces, a
public one and private one (connecting to a local network).
auto eth0
iface eth0 inet manual
auto eth1
iface eth1 inet static
address 10.158.231.29
netmask 255.255.255.0
network 10.158.231.0
broadcast 10.158.231.255
gateway 10.158.231.1
auto cloudbr0
iface cloudbr0 inet static
address 172.16.8.7
netmask 255.255.255.0
network 172.16.8.0
broadcast 172.16.8.255
up route add -net 172.16.0.0 netmask 255.255.0.0 gw 172.16.8.1
bridge_ports eth0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
The idea is to connect all the guests to the 172.16 network because access to
multicasting devices is needed.
The systems are running on Debian wheezy, and got the system up after fixing
the /etc/legal problem (echo Ubuntu) and after finding out in the logs that
there is a package dependency problem on the cloudstack-manager package for 4.3
The biggest problem seems to be that host is running in a corporate network and
cloudstack is auto configuring some things by downloading over http.
This does not work since a corporate proxy is required.
I downloaded the system images manually; but now it seems I need to do the same
in the secondary storage vm.
The vm is running and I can get the login prompt with virsh (kvm), however
there is no password I can find.
The trick with the connection to the local-link address does not work because I
fear I have a networking/firewall issue. It does not matter if I disable the
firewall or not, I cannot access the 169.254.x.x network.
pinging the device returns
ping 169.254.3.236
PING 169.254.3.236 (169.254.3.236) 56(84) bytes of data.
From 169.254.0.1 icmp_seq=1 Destination Host Unreachable From 169.254.0.1
icmp_seq=2 Destination Host Unreachable From 169.254.0.1 icmp_seq=3 Destination
Host Unreachable
The firewall configuration is firehol
cat /etc/firehol/firehol.conf
version 5
# Accept all client traffic on any interface
FIREHOL_LOG_MODE="ULOG"
server_cloudstackweb_ports="tcp/8080"
client_cloudstackweb_ports="default"
server_buildbot_ports="tcp/8010"
client_buildbot_ports="default"
server_git_ports="tcp/9418"
client_git_ports="default"
labo_ips="172.16.0.0/16 169.254.0.0/16"
server_cloudstack_ports="tcp/1798"
client_cloudstack_ports="default"
server_libvirt_ports="tcp/16509"
client_libvirt_ports="default"
server_vnc_ports="tcp/5900:6100"
client_vnc_ports="default"
server_libvirtlive_ports="tcp/49152:49216"
client_libvirtlive_ports="default"
# local bridge address
interface cloudbr0 LAN src "${labo_ips}"
server all accept
client all accept
interface eth1 WAN src not "${labo_ips}"
protection strong
policy reject
server ssh accept
server cloudstack accept
server cloudstackweb accept
server libvirt accept
server vnc accept
server libvirtlive accept
server buildbot accept
client all accept
# zeroconf bridge address
interface cloud0 LLBR0
client all accept
server all accept
router LAN2WAN inface cloudbr0 outface eth1
masquerade
route all accept
I got a bit further by adding the interface cloud0 to the definition, but still
no joy.
I need to access the vm to run some script with a http_proxy environment
variable...
I am currently starting to run out of ideas; installing cloudstack was fine;
but I entered hell afterwards; ...
/sbin/ifconfig
cloud0Link encap:Ethernet HWaddr fe:00:a9:fe:01:f8
inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
inet6 addr: fe80::4061:7aff:fe05:f240/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:4278
Re: firewall problems and proxy problems
On 16 October 2014 17:53, Daan Hoogland wrote: > On Thu, Oct 16, 2014 at 5:33 PM, Marc Leeman > wrote: > > > Is there some inconsistency in the database? > > > This might well be. To check this look at mshost and mshost_peer. > > Are you upgrading? Is this a test setup? > > > It is my first setup :-) mysql> select * from mshost_peer; ++--+-+---++-+ | id | owner_mshost | peer_mshost | peer_runid| peer_state | last_update | ++--+-+---++-+ | 1 |1 | 1 | 1413300325989 | Up | 2014-10-14 15:25:55 | | 2 |2 | 2 | 1413301475290 | Up | 2014-10-14 15:44:48 | | 16 |3 | 3 | 1413473252587 | Up | 2014-10-16 15:27:45 | ++--+-+---++-+ 3 rows in set (0.00 sec) mysql> select * from mshost; ++-+---+--+---+-++--+-+-+-+ | id | msid| runid | name | state | version | service_ip | service_port | last_update | removed | alert_count | ++-+---+--+---+-++--+-+-+-+ | 1 | 172136268721636 | 1413300325989 | zee | Up| 4.3.1 | 172.16.8.7 | 9090 | 2014-10-14 15:41:40 | NULL| 0 | | 2 | 72986426470976 | 1413301475290 | zee | Up| 4.3.1 | 172.16.8.7 | 9090 | 2014-10-14 16:18:24 | NULL| 0 | | 3 | 279278805451256 | 1413473252587 | zee | Up| 4.3.1 | 172.16.8.7 | 9090 | 2014-10-17 08:37:41 | NULL| 0 | ++-+---+--+---+-++--+-+-+-+ 3 rows in set (0.00 sec)
Re: firewall problems and proxy problems
ok, then I'd re-init the db and try again, Keeping Pauls remark in mind. I usually mess up my ip space and find myself stuck in a mess. Paul didn't you post a sample layout somewhere? On Fri, Oct 17, 2014 at 10:38 AM, Marc Leeman wrote: > On 16 October 2014 17:53, Daan Hoogland wrote: > > > On Thu, Oct 16, 2014 at 5:33 PM, Marc Leeman > > wrote: > > > > > Is there some inconsistency in the database? > > > > > This might well be. To check this look at mshost and mshost_peer. > > > > Are you upgrading? Is this a test setup? > > > > > > > It is my first setup :-) > > mysql> select * from mshost_peer; > > ++--+-+---++-+ > | id | owner_mshost | peer_mshost | peer_runid| peer_state | > last_update | > > ++--+-+---++-+ > | 1 |1 | 1 | 1413300325989 | Up | 2014-10-14 > 15:25:55 | > | 2 |2 | 2 | 1413301475290 | Up | 2014-10-14 > 15:44:48 | > | 16 |3 | 3 | 1413473252587 | Up | 2014-10-16 > 15:27:45 | > > ++--+-+---++-+ > 3 rows in set (0.00 sec) > > mysql> select * from mshost; > > ++-+---+--+---+-++--+-+-+-+ > | id | msid| runid | name | state | version | > service_ip | service_port | last_update | removed | alert_count | > > ++-+---+--+---+-++--+-+-+-+ > | 1 | 172136268721636 | 1413300325989 | zee | Up| 4.3.1 | > 172.16.8.7 | 9090 | 2014-10-14 15:41:40 | NULL| 0 | > | 2 | 72986426470976 | 1413301475290 | zee | Up| 4.3.1 | > 172.16.8.7 | 9090 | 2014-10-14 16:18:24 | NULL| 0 | > | 3 | 279278805451256 | 1413473252587 | zee | Up| 4.3.1 | > 172.16.8.7 | 9090 | 2014-10-17 08:37:41 | NULL| 0 | > > ++-+---+--+---+-++--+-+-+-+ > 3 rows in set (0.00 sec) > -- Daan
Re: firewall problems and proxy problems
I cleared the database and re-did the installation and all seems to be working now. Thanks for the help. It seems that something must have been off during the initial install.
Re: firewall problems and proxy problems
On Oct 17, 2014, at 11:01 AM, Marc Leeman wrote: > I cleared the database and re-did the installation and all seems to be > working now. > > Thanks for the help. > > It seems that something must have been off during the initial install. Hi Marc, I was at the guy at the booth :) In my angst to get things working I probably messed up the installed when upgrading rather quickly to 4.3. In any case, glad it's working now, please confirm that you can launch instances. I have kept track of your bugs for debian packaging and we will try to address those. -sebastien
Re: firewall problems and proxy problems
> I cleared the database and re-did the installation and all seems to be > > working now. > > > > Thanks for the help. > > > > It seems that something must have been off during the initial install. > > Hi Marc, I was at the guy at the booth :) > Yeah, thank you; you were very patient. Thank you for helping me out in Cloudstack; it was pretty new to me :-) > In my angst to get things working I probably messed up the installed when > upgrading rather quickly to 4.3. > > In any case, glad it's working now, please confirm that you can launch > instances. > Truth be said, the config I am running now is probably not what I was initially looking but I thought go for the out-of-the-box installation and take it from there. I have assigned the internal addresses to be a range of the external network, but I noticed that though I see the connections from the VMs coming from the correct IP (when e.g. ssh-ing to an external server); but I cannot connect back (ingress rules are very strict). Is there any resource you'd recommend to convert the basic guest network to a configuration where all the VMs are exposed to the network as if they were machines sitting on that network? One thing I've also noticed when deploying test software on the cloud is that I cannot use DHCP: The machines are installed by default with a static IP (software needs to be predictable and on the network). When changing over the static IP to a DHCP address; I see that the system gets a DHCP address that seems (need to verify this, but I would need to check the DHCP server on the switch to which I do not have access atm) to come from the network. However the system is muted on the network. Only when I set the IP to the IP it got assigned by cloudstack can I access the network. > I have kept track of your bugs for debian packaging and we will try to > address those. > >
Re: firewall problems and proxy problems
Just some final feedback on the setup I had. I have an installation up and running since about a week ago. A number of remarks I got from Sebastien allowed me to put things in perspective (especially when trying to get the network configured). All in all, a pretty standard configuration seems to be working for me (environment where the VMs are directly accessible on the same network as the host), mainly for streaming (multicast) testing. The main problems I had were: 1. Running it on GNU/Debian is not really a problem, but due to an in complete dependency in the package (entered a bug for that one); a binary was not present and that caused a bit of a problem. I assume that the binary gets pulled in in Ubuntu in some other way if no-one had a problem with that one. Adding a dependency should fix the problem just fine with no side effects. Even though the package is to my knowledge not in Debian, it is pretty easy enough to pull it in from Ubuntu and install it (at the very least, a user will get a heads up about the missing program (script btw)). 2. A reset of the installation is tricky to do and is not offered (to my knowledge) in the system itself. This caused me to use a purge script to clear the data bases and reset the system. On top of that (and that one is on me); I forgot to change a password in the script that made my system even more corrupt than it was. Finally; it was a remark of Daan that pointed out the problem to me (not the root cause, but once I noticed the inconsistency in the DB Daan pointed out, it was pretty clear what could have been the culprit). mleeman@zee:~$ cat bin/cloudstack-reset.sh #!/bin/sh /etc/init.d/cloudstack-management stop mysql -p -e 'drop database cloud' mysql -p -e 'drop database cloud_usage' cloudstack-setup-databases cloud:@localhost --deploy-as=root: rm -rf /var/log/cloud/management/* cloudstack-setup-management /etc/init.d/cloudstack-management start 3. What really caused a problem and confusion is that I was not aware that, during the installation part of the install is downloaded. That in itself is not a problem; but the combination that there is no feedback on this process AND that my system is in an environment where the use of a proxy is mandatory; let the install fail silently... After someone dropped a remark on the 'download during install'; I pieced it together and started logging in to the systems and by setting the http_proxy environment variable, I got everything up and running. Thanks for all the pointers, the help and the patience!
