Re: VirtualRouter has duplicated public interface and broken NAT functionality

[Remi Bergsma]

Hi,

You are right, eth3 should not be there. Somehow the detection of the public 
interface goes wrong and it ends up creating a new one.

In the logs, instead of ethX, I see null which is an indication:

2015-11-17 01:38:17,760 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.IpAssocCommand
2015-11-17 01:38:17,769 DEBUG [kvm.resource.OvsVifDriver] 
(agentRequest-Handler-1:null) plugging nic=[Nic:Public-null-vlan://104]

I've seen this before and I think it is the combination of ovs with a tagged 
(as in vlan) network. 

You probably have specified a vlan tag in CloudStack (104?) on some bridge. If 
you can change it to use untagged, but then point to an ovs bridge that is 
tagged it will most likely work. You then move the tagging from CloudStack to 
ovs. I think this is how I worked around it when I saw this. 

Don't get me wrong, what you did should work so this is a bug. The cause needs 
to be figured out but it wasn't too obvious. 

I'm also curious to know if 4.6.0 (which will be available this week) still has 
this issue. 

This is what I remember, hope it gives some pointers. 

Regards, Remi 



> On 17 Nov 2015, at 06:56, Alexander Couzens  wrote:
> 
> Hi,
> 
> a VirtualRouter is not nat-ing any Packets. Even there are egress
> rules, it's not possible to ping anything from a guest VM to the outer world.
> 
> I'm not sure if I'm doing something wrong or if this is a bug.
> 
> The VR has XXX.XXX.221.102 as public ip, guest net is 10.1.1.0/24.
> The webui + cloudmonkey shows me 3 nics on the VM, but it has obvious 4 nics.
> Looking into the database confirms it, there isn't any forth nic.
> 
> Logging into the machine shows
> # ip r
> default via XXX.XXX.221.126 dev eth2 
> 10.1.1.0/24 dev eth0  proto kernel  scope link  src 10.1.1.1 
> XXX.XXX.221.96/27 dev eth2  proto kernel  scope link  src XXX.XXX.221.102 
> XXX.XXX.221.96/27 dev eth3  proto kernel  scope link  src XXX.XXX.221.102 
> 169.254.0.0/16 dev eth1  proto kernel  scope link  src 169.254.0.158 
> 
> Also looking on iptables I found the nat rule:
> Chain POSTROUTING (policy ACCEPT 6 packets, 415 bytes)
> pkts bytes target prot opt in out source   
> destination 
>0 0 SNAT   all  --  *  eth30.0.0.0/0 0.0.0.0/0
> to:XXX.XXX.221.102
> 
> Mention the eth3, which is the later interface. But the traffic will route 
> from the guest vm is
> using eth2. There are policy routing which uses the table Table_eth3, but it 
> also uses eth2 for traffic instead of eth3.
> # ip r s ta Table_eth3
> default via XXX.XXX.221.126 dev eth2  proto static 
> throw 10.1.1.0/24  proto static 
> throw XXX.XXX.221.96/27  proto static 
> throw 169.254.0.0/16  proto static 
> 
> So I would say that eth3 is wrong.
> When the vm is created it only has 3 network interface and looks correct, but 
> later cloudstack
> generates the last one looking into the log [1].
> 
> Any ideas?
> 
> Best
> lynxis
> 
> 
> [1]
> 2015-11-17 01:38:17,760 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.AggregationControlCommand
> 2015-11-17 01:38:17,760 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.IpAssocCommand
> 2015-11-17 01:38:17,769 DEBUG [kvm.resource.OvsVifDriver] 
> (agentRequest-Handler-1:null) plugging nic=[Nic:Public-null-vlan://104]
> 2015-11-17 01:38:17,898 DEBUG [kvm.resource.LibvirtComputingResource] 
> (agentRequest-Handler-1:null) Executing: 
> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 
> 169.254.0.158 -a eth3 
> 2015-11-17 01:38:18,033 DEBUG [kvm.resource.LibvirtComputingResource] 
> (agentRequest-Handler-1:null) Execution is successful.
> 2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.SetFirewallRulesCommand
> 2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.SetMonitorServiceCommand
> 2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.DhcpEntryCommand
> 2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.VmDataCommand
> 2015-11-17 01:38:18,035 DEBUG [cloud.agent.Agent] 
> (agentRequest-Handler-1:null) Processing command: 
> com.cloud.agent.api.routing.AggregationControlCommand
> 2015-11-17 01:38:18,271 DEBUG [kvm.resource.LibvirtComputingResource] 
> (agentRequest-Handler-1:null) Executing: 
> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 
> 169.254.0.158 -c /var/cache/cloud/VR-1c6cc724-79ac-4155-bb73-284514213d10.cfg 
> 2015-11-17 01:38:19,062 DEBUG [kvm.resource.LibvirtComputingResource] 
> (agentRequest-Handler-1:null) Execution i

VirtualRouter has duplicated public interface and broken NAT functionality

[Alexander Couzens]

Hi,

a VirtualRouter is not nat-ing any Packets. Even there are egress
rules, it's not possible to ping anything from a guest VM to the outer world.

I'm not sure if I'm doing something wrong or if this is a bug.

The VR has XXX.XXX.221.102 as public ip, guest net is 10.1.1.0/24.
The webui + cloudmonkey shows me 3 nics on the VM, but it has obvious 4 nics.
Looking into the database confirms it, there isn't any forth nic.

Logging into the machine shows
# ip r
default via XXX.XXX.221.126 dev eth2 
10.1.1.0/24 dev eth0  proto kernel  scope link  src 10.1.1.1 
XXX.XXX.221.96/27 dev eth2  proto kernel  scope link  src XXX.XXX.221.102 
XXX.XXX.221.96/27 dev eth3  proto kernel  scope link  src XXX.XXX.221.102 
169.254.0.0/16 dev eth1  proto kernel  scope link  src 169.254.0.158 

Also looking on iptables I found the nat rule:
Chain POSTROUTING (policy ACCEPT 6 packets, 415 bytes)
 pkts bytes target prot opt in out source   destination 

0 0 SNAT   all  --  *  eth30.0.0.0/0 0.0.0.0/0
to:XXX.XXX.221.102

Mention the eth3, which is the later interface. But the traffic will route from 
the guest vm is
using eth2. There are policy routing which uses the table Table_eth3, but it 
also uses eth2 for traffic instead of eth3.
# ip r s ta Table_eth3
default via XXX.XXX.221.126 dev eth2  proto static 
throw 10.1.1.0/24  proto static 
throw XXX.XXX.221.96/27  proto static 
throw 169.254.0.0/16  proto static 

So I would say that eth3 is wrong.
When the vm is created it only has 3 network interface and looks correct, but 
later cloudstack
generates the last one looking into the log [1].

Any ideas?

Best
lynxis


[1]
2015-11-17 01:38:17,760 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.AggregationControlCommand
2015-11-17 01:38:17,760 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.IpAssocCommand
2015-11-17 01:38:17,769 DEBUG [kvm.resource.OvsVifDriver] 
(agentRequest-Handler-1:null) plugging nic=[Nic:Public-null-vlan://104]
2015-11-17 01:38:17,898 DEBUG [kvm.resource.LibvirtComputingResource] 
(agentRequest-Handler-1:null) Executing: 
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 
169.254.0.158 -a eth3 
2015-11-17 01:38:18,033 DEBUG [kvm.resource.LibvirtComputingResource] 
(agentRequest-Handler-1:null) Execution is successful.
2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.SetFirewallRulesCommand
2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.SetMonitorServiceCommand
2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.DhcpEntryCommand
2015-11-17 01:38:18,034 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.VmDataCommand
2015-11-17 01:38:18,035 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) 
Processing command: com.cloud.agent.api.routing.AggregationControlCommand
2015-11-17 01:38:18,271 DEBUG [kvm.resource.LibvirtComputingResource] 
(agentRequest-Handler-1:null) Executing: 
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 
169.254.0.158 -c /var/cache/cloud/VR-1c6cc724-79ac-4155-bb73-284514213d10.cfg 
2015-11-17 01:38:19,062 DEBUG [kvm.resource.LibvirtComputingResource] 
(agentRequest-Handler-1:null) Execution is successful.

## versions and setup
using ubuntu 14.04 machines with packages:
cloudstack-management 4.5.2
cloudstack-agent 4.5.2 using kvm

system vm:
Cloudstack Release 4.5.2 Tue Aug 11 00:42:47 UTC 2015

network configuration
advanced setup, with management, guest, internet
# kvm-label - descriptions
cloudbr0 - management.
ovs-trunk - guest,internet. trunk interface using openvswitch + vlans

As primary storage I'm using a shared-mountpoint,
As secondary storage I'm using nfs over the management network.

consolenproxy + storage vm are working.
-- 
Alexander Couzens

mail: lyn...@fe80.eu
jabber: lyn...@fe80.eu
mobile: +4915123277221
gpg: 390D CF78 8BF9 AA50 4F8F  F1E2 C29E 9DA6 A0DF 8604


pgpl6upQLcO0h.pgp
Description: OpenPGP digital signature

Re: gpg verification: missing key 0EE3D884

[John Kinsella]

Rohit - looks like your key isn’t in 
https://dist.apache.org/repos/dist/release/cloudstack/KEYS ?

On Nov 16, 2015, at 3:43 PM, Udo Rader 
mailto:list...@bestsolution.at>> wrote:

Hi,

I've downloaded the latest 4.5.2 tar.bz2 and tried to verify the
download using gpg, but gpg tells me that the used key is unknown:

[udo@artio Downloads]$ gpg --verify apache-cloudstack-4.5.2-src.tar.bz2.asc
gpg: assuming signed data in `apache-cloudstack-4.5.2-src.tar.bz2'
gpg: Signature made Wed 19 Aug 2015 11:13:04 AM CEST using RSA key ID
0EE3D884
gpg: Can't check signature: public key not found

So is the key missing from http://www.apache.org/dist/cloudstack/KEYS or
am I missing something?

Regards

Udo

gpg verification: missing key 0EE3D884

[Udo Rader]

Hi,

I've downloaded the latest 4.5.2 tar.bz2 and tried to verify the
download using gpg, but gpg tells me that the used key is unknown:

[udo@artio Downloads]$ gpg --verify apache-cloudstack-4.5.2-src.tar.bz2.asc
gpg: assuming signed data in `apache-cloudstack-4.5.2-src.tar.bz2'
gpg: Signature made Wed 19 Aug 2015 11:13:04 AM CEST using RSA key ID
0EE3D884
gpg: Can't check signature: public key not found

So is the key missing from http://www.apache.org/dist/cloudstack/KEYS or
am I missing something?

Regards

Udo

Connection issue between host and server

[Yan Bai]

Hello,

I am experiencing an connection issue between host and server. I am using
Cloudstack 4.4.

Below is the log found on host:

2015-11-16 16:46:54,448 ERROR [utils.nio.NioConnection]
(Agent-Selector:null) Unable to initialize the threads.
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
closed with -1 on reading size.
at com.cloud.utils.nio.NioClient.init(NioClient.java:90)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:113)
at java.lang.Thread.run(Thread.java:745)
2015-11-16 16:46:59,458 INFO  [utils.nio.NioClient] (Agent-Selector:null)
Connecting to 192.168.1.10:8250
2015-11-16 16:47:29,495 ERROR [utils.nio.NioConnection]
(Agent-Selector:null) Unable to initialize the threads.
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
closed with -1 on reading size.
at com.cloud.utils.nio.NioClient.init(NioClient.java:90)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:113)
at java.lang.Thread.run(Thread.java:745)
2015-11-16 16:47:34,497 INFO  [utils.nio.NioClient] (Agent-Selector:null)
Connecting to 192.168.1.10:8250
2015-11-16 16:48:04,538 ERROR [utils.nio.NioConnection]
(Agent-Selector:null) Unable to initialize the threads.
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
closed with -1 on reading size.
at com.cloud.utils.nio.NioClient.init(NioClient.java:90)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:113)
at java.lang.Thread.run(Thread.java:745)
2015-11-16 16:48:09,540 INFO  [utils.nio.NioClient] (Agent-Selector:null)
Connecting to 192.168.1.10:8250
2015-11-16 16:48:39,576 ERROR [utils.nio.NioConnection]
(Agent-Selector:null) Unable to initialize the threads.
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
closed with -1 on reading size.
at com.cloud.utils.nio.NioClient.init(NioClient.java:90)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:113)
at java.lang.Thread.run(Thread.java:745)


192.168.1.10 is ip address of management server. Host ip address is
192.168.1.11. I tried "telnet 192.168.1.10 8250" and it works fine.

There is a ticket on JIRA about this issue shows that this issue has been
fixed.

Thanks,
Yan

Re: CloudStack meetup last week

[Özhan Rüzgar Karaman]

Hi Steve;
Its great that videos are available for us who could not attend to the
event, thanks to all contributors.

Regards
Özhan

On Mon, Nov 16, 2015 at 6:16 PM, Steve Roles 
wrote:

> Hi all – if you’re interested please see here for a roundup of last week’s
> CloudStack European User Group meet here in London:
> http://www.shapeblue.com/cloudstack-european-user-group-roundup-november-2015
>
>
>
> Thanks to Rene Moser, Daan Hoogland, Jon Noble and Paul Angus for talking.
>
>
>
> Regards,
>
>
>
> Steve Roles
>
> *VP Service Delivery*
>
>
>
>
>
> D: +44 20 3642 6102 <+442036426102> | S: +44 20 3603 0540 <+442036030540>
> | M: +44 7770 745 036 <+447770745036>
>
>
>
> steve.ro...@shapeblue.com | www.shapeblue.com | Twitter:@ShapeBlue
> 
>
>
>
> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
>
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build
> 
> CSForge – rapid IaaS deployment framework 
> CloudStack Consulting 
> CloudStack Software Engineering
> 
> CloudStack Infrastructure Support
> 
> CloudStack Bootcamp Training Courses
> 
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>

CloudStack meetup last week

[Steve Roles]

Hi all - if you're interested please see here for a roundup of last week's 
CloudStack European User Group meet here in London: 
http://www.shapeblue.com/cloudstack-european-user-group-roundup-november-2015

Thanks to Rene Moser, Daan Hoogland, Jon Noble and Paul Angus for talking.

Regards,

Steve Roles
VP Service Delivery

[cid:image002.png@01D1208A.15085BA0]

D: +44 20 3642 6102 | S: +44 20 3603 0540 
| M: +44 7770 745 036

steve.ro...@shapeblue.com | 
www.shapeblue.com | 
Twitter:@ShapeBlue

ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build
CSForge - rapid IaaS deployment framework
CloudStack Consulting
CloudStack Software 
Engineering
CloudStack Infrastructure 
Support
CloudStack Bootcamp Training Courses

This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Ltd or related companies. If you are not the 
intended recipient of this email, you must neither take any action based upon 
its contents, nor copy or show it to anyone. Please contact the sender if you 
believe you have received this email in error. Shape Blue Ltd is a company 
incorporated in England & Wales. ShapeBlue Services India LLP is a company 
incorporated in India and is operated under license from Shape Blue Ltd. Shape 
Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company 
registered by The Republic of South Africa and is traded under license from 
Shape Blue Ltd. ShapeBlue is a registered trademark.