Hello all, There is some bug after upgrade from 4.1.1 to 4.3.0 KVM Hypervizor
Agent settings labels : guest.network.device=cloudbr1 private.network.device=cloudbr1 public.network.device=cloudbr0 After upgrade to 4.3 SSVM, all VR's started with multiple [public] interfaces, I have some VR's with 5-6 ethX interfaces with same IP So, in all cases egress rules doesn't work. Fix? Workaround? On Sat, Apr 12, 2014 at 12:16 AM, motty cruz <motty.c...@gmail.com> wrote: > I have a testing cloudstack cluster, I destroyed it and rebuilding upgraded > serveral times, each time I ran into the same problem, unable to access > outside world from instances behind virtual router. > > here is iptables before upgrade, Cloudstack 4.2 > > # Generated by iptables-save v1.4.14 on Fri Apr 11 19:53:57 2014 > *mangle > :PREROUTING ACCEPT [2317:1282555] > :INPUT ACCEPT [409:147015] > :FORWARD ACCEPT [0:0] > :OUTPUT ACCEPT [189:29312] > :POSTROUTING ACCEPT [189:29312] > :FIREWALL_176.23.23.192 - [0:0] > :VPN_176.23.23.192 - [0:0] > -A PREROUTING -d 176.23.23.192/32 -j VPN_176.23.23.192 > -A PREROUTING -d 176.23.23.192/32 -j FIREWALL_176.23.23.192 > -A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK > --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff > -A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill > -A FIREWALL_176.23.23.192 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FIREWALL_176.23.23.192 -j DROP > -A VPN_176.23.23.192 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A VPN_176.23.23.192 -j RETURN > COMMIT > # Completed on Fri Apr 11 19:53:57 2014 > # Generated by iptables-save v1.4.14 on Fri Apr 11 19:53:57 2014 > *filter > :INPUT DROP [204:117504] > :FORWARD DROP [0:0] > :OUTPUT ACCEPT [150:22404] > :FW_OUTBOUND - [0:0] > :NETWORK_STATS - [0:0] > -A INPUT -j NETWORK_STATS > -A INPUT -d 224.0.0.18/32 -j ACCEPT > -A INPUT -d 225.0.0.50/32 -j ACCEPT > -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A INPUT -p icmp -j ACCEPT > -A INPUT -i lo -j ACCEPT > -A INPUT -i eth0 -p udp -m udp --dport 67 -j ACCEPT > -A INPUT -i eth0 -p udp -m udp --dport 53 -j ACCEPT > -A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT > -A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 3922 -j ACCEPT > -A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT > -A INPUT -s 10.1.1.0/24 -i eth0 -p tcp -m state --state NEW -m tcp --dport > 8080 -j ACCEPT > -A FORWARD -j NETWORK_STATS > -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth0 -o eth0 -m state --state NEW -j ACCEPT > -A FORWARD -i eth0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth0 -o eth2 -j FW_OUTBOUND > -A OUTPUT -j NETWORK_STATS > -A FW_OUTBOUND -m state --state RELATED,ESTABLISHED -j ACCEPT > -A NETWORK_STATS -i eth0 -o eth2 > -A NETWORK_STATS -i eth2 -o eth0 > -A NETWORK_STATS ! -i eth0 -o eth2 -p tcp > -A NETWORK_STATS -i eth2 ! -o eth0 -p tcp > COMMIT > # Completed on Fri Apr 11 19:53:57 2014 > # Generated by iptables-save v1.4.14 on Fri Apr 11 19:53:57 2014 > *nat > :PREROUTING ACCEPT [2078:1204416] > :INPUT ACCEPT [10:964] > :OUTPUT ACCEPT [1:338] > :POSTROUTING ACCEPT [1:338] > -A POSTROUTING -o eth2 -j SNAT --to-source 176.23.23.192 > COMMIT > # Completed on Fri Apr 11 19:53:57 2014 > > > after upgrading to Cloudstack 4.3 > > > :POSTROUTING ACCEPT [211:25828] > :FIREWALL_176.23.23.192 - [0:0] > :VPN_176.23.23.192 - [0:0] > -A PREROUTING -d 176.23.23.192/32 -j VPN_176.23.23.192 > -A PREROUTING -d 176.23.23.192/32 -j FIREWALL_176.23.23.192 > -A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK > --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff > -A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill > -A FIREWALL_176.23.23.192 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FIREWALL_176.23.23.192 -j DROP > -A VPN_176.23.23.192 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A VPN_176.23.23.192 -j RETURN > COMMIT > # Completed on Fri Apr 11 20:49:46 2014 > # Generated by iptables-save v1.4.14 on Fri Apr 11 20:49:46 2014 > *filter > :INPUT DROP [68:32168] > :FORWARD DROP [0:0] > :OUTPUT ACCEPT [81:12516] > :FW_EGRESS_RULES - [0:0] > :FW_OUTBOUND - [0:0] > :NETWORK_STATS - [0:0] > -A INPUT -j NETWORK_STATS > -A INPUT -d 224.0.0.18/32 -j ACCEPT > -A INPUT -d 225.0.0.50/32 -j ACCEPT > -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A INPUT -p icmp -j ACCEPT > -A INPUT -i lo -j ACCEPT > -A INPUT -i eth0 -p udp -m udp --dport 67 -j ACCEPT > -A INPUT -i eth0 -p udp -m udp --dport 53 -j ACCEPT > -A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT > -A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 3922 -j ACCEPT > -A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT > -j ACCEPT > -A FORWARD -j NETWORK_STATS > -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth0 -o eth0 -m state --state NEW -j ACCEPT > -A FORWARD -i eth0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth0 -o eth2 -j FW_OUTBOUND > -A FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FORWARD -i eth0 -o eth3 -j FW_OUTBOUND > -A OUTPUT -j NETWORK_STATS > -A FW_EGRESS_RULES -p udp -m udp --dport 1:65535 -j ACCEPT > -A FW_EGRESS_RULES -p icmp -m icmp --icmp-type 8/0 -j ACCEPT > -A FW_EGRESS_RULES -p tcp -m tcp --dport 1:65535 -j ACCEPT > -A FW_OUTBOUND -m state --state RELATED,ESTABLISHED -j ACCEPT > -A FW_OUTBOUND -j FW_EGRESS_RULES > -A NETWORK_STATS -i eth0 -o eth2 > -A NETWORK_STATS -i eth2 -o eth0 > -A NETWORK_STATS ! -i eth0 -o eth2 -p tcp > -A NETWORK_STATS -i eth2 ! -o eth0 -p tcp > -A NETWORK_STATS -i eth0 -o eth3 > -A NETWORK_STATS -i eth3 -o eth0 > -A NETWORK_STATS ! -i eth0 -o eth3 -p tcp > -A NETWORK_STATS -i eth3 ! -o eth0 -p tcp > COMMIT > # Completed on Fri Apr 11 20:49:46 2014 > # Generated by iptables-save v1.4.14 on Fri Apr 11 20:49:46 2014 > *nat > :PREROUTING ACCEPT [685:441108] > :INPUT ACCEPT [7:492] > :OUTPUT ACCEPT [2:136] > :POSTROUTING ACCEPT [3:412] > -A POSTROUTING -o eth3 -j SNAT --to-source 199.96.39.192 > COMMIT > # Completed on Fri Apr 11 20:49:46 2014 > > > Please help!! is a KVM cluster, I have two bridges, cloudbr0 and cloudbr1. > bridge cloudbr0 is default public/guest and cloudbr1 is management. > > I am able to ping from virtual router, I personally think source NAT is the > issue, > > Thanks, > > > On Fri, Apr 11, 2014 at 8:14 AM, motty cruz <motty.c...@gmail.com> wrote: > > > looking at the VR it has three interfaces NIC1 10.1.1.1 NIC2 (type > > control) Local Link NIC3(default NIC) public IP > > > > when I log in to the router and ran the following command: > > ifconfig -a > > I see 4 interfaces eth0 10.1.1.1 eth1 (local link)169.254.3.81 eth2 > public > > IP eth3 (same public IP as eth2). > > > > I believe this is my problem, but don't know how to fix it, > > any ideas? > > > > > > On Fri, Apr 11, 2014 at 8:04 AM, motty cruz <motty.c...@gmail.com> > wrote: > > > >> Thanks Suresh, > >> > >> I tried your suggestion and I'm not able to access outside the VR > router. > >> I am stumped! > >> > >> please help! > >> > >> > >> > >> > >> On Thu, Apr 10, 2014 at 7:21 AM, Suresh Sadhu <suresh.sa...@citrix.com > >wrote: > >> > >>> Ok then work around is manually append rule to cloudbr1 . > >>> > >>> Take the backup of iptables rules > >>> Manfully detach the eth interface from cloudbr0 and attach to > cloudbr1 > >>> Apply the all exiting firewall rules manually on the interface gain > >>> > >>> > >>> After that your VMs will access the public network. > >>> > >>> > >>> Regards > >>> Sadhu > >>> > >>> > >>> > >>> -----Original Message----- > >>> From: motty cruz [mailto:motty.c...@gmail.com] > >>> Sent: 10 April 2014 19:40 > >>> To: users@cloudstack.apache.org > >>> Subject: Re: Cloudstack 4.3 instances can't access outside world > >>> > >>> yes, I'm am using traffic labels, everything was working fine before > the > >>> upgrade to 4.3. did not change anything on the cloudbr0 or cloudbr1. > >>> > >>> > >>> On Thu, Apr 10, 2014 at 7:05 AM, Suresh Sadhu <suresh.sa...@citrix.com > >>> >wrote: > >>> > >>> > Did you used traffic name labels? > >>> > > >>> > In 4.3 traffic labels are not considering ,by default its attaching > to > >>> > default traffic labels(eg:in KVM its cloudbr0 ...due to this unable > >>> > to access public network i.r before upgrade if ieth2 attached > cloudbr1 > >>> > and after upgrade its attached to cloudbr0).maybe you are hitting > this > >>> issue. > >>> > > >>> > Regards > >>> > sadhu > >>> > > >>> > > >>> > -----Original Message----- > >>> > From: motty cruz [mailto:motty.c...@gmail.com] > >>> > Sent: 10 April 2014 19:28 > >>> > To: users@cloudstack.apache.org > >>> > Subject: Re: Cloudstack 4.3 instances can't access outside world > >>> > > >>> > yes I can ping VR, also after the upgrade VR has four insterfaces, > >>> > eth0 subnet for Instances, eth1, eth2 for public IP and eth3 for > >>> public IP. > >>> > > >>> > > >>> > On Wed, Apr 9, 2014 at 10:35 PM, Erik Weber <terbol...@gmail.com> > >>> wrote: > >>> > > >>> > > Can you ping the VR? Log on to the VR, and get the iptables rules. > >>> > > How do they look? > >>> > > > >>> > > Erik Weber > >>> > > 10. apr. 2014 00:21 skrev "motty cruz" <motty.c...@gmail.com> > >>> følgende: > >>> > > > >>> > > > I did add egress rules, reboot network but no sucess, so I > removed > >>> > > > that rules and nothing. > >>> > > > > >>> > > > I am lost. > >>> > > > > >>> > > > > >>> > > > On Wed, Apr 9, 2014 at 9:08 AM, Erik Weber <terbol...@gmail.com> > >>> > wrote: > >>> > > > > >>> > > > > Did you remove the egress rule again? If not, try that. > >>> > > > > > >>> > > > > Erik > >>> > > > > 9. apr. 2014 15:49 skrev "motty cruz" <motty.c...@gmail.com> > >>> > følgende: > >>> > > > > > >>> > > > > > yes I try adding the rule, restart network and router but no > >>> > success! > >>> > > > > > > >>> > > > > > > >>> > > > > > On Tue, Apr 8, 2014 at 11:16 PM, Erik Weber > >>> > > > > > <terbol...@gmail.com> > >>> > > > wrote: > >>> > > > > > > >>> > > > > > > Try adding an egress rule, and removing it again. > >>> > > > > > > > >>> > > > > > > We experience the same, but has so far believed it was > >>> > > > > > > because we > >>> > > > > changed > >>> > > > > > > the default rule from deny to allow after accounts were > >>> made.. > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > On Tue, Apr 8, 2014 at 11:14 PM, motty cruz > >>> > > > > > > <motty.c...@gmail.com> > >>> > > > > > wrote: > >>> > > > > > > > >>> > > > > > > > I have two isolated network both virtual routers can ping > >>> > > anywhere, > >>> > > > > but > >>> > > > > > > the > >>> > > > > > > > Instances behind the virtual router can't ping or access > >>> > > > > > > > the > >>> > > > > internet. > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > On Tue, Apr 8, 2014 at 10:38 AM, motty cruz < > >>> > > motty.c...@gmail.com> > >>> > > > > > > wrote: > >>> > > > > > > > > >>> > > > > > > > > Hello, > >>> > > > > > > > > I'm having issues with VMs unable to access outside > >>> world. > >>> > > > > > > > > I > >>> > > can > >>> > > > > ping > >>> > > > > > > > > gateway, also when I log in to virtual router, I am > able > >>> > > > > > > > > to > >>> > > ping > >>> > > > > > > > > google.com or anywhere. > >>> > > > > > > > > in the Egress rules I am allowing all. reboot network > >>> > > > > > > > > and > >>> > > virtual > >>> > > > > > > router > >>> > > > > > > > > does not help. > >>> > > > > > > > > > >>> > > > > > > > > VMs were able to access outside before upgrading from > >>> > > > > > > > > 4.2 to > >>> > > 4.3. > >>> > > > > > > > > > >>> > > > > > > > > any ideas? > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >> > >> > > > -- ttyv0 "/usr/libexec/gmail Pc" webcons on secure
