[ https://issues.apache.org/jira/browse/CLOUDSTACK-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744681#comment-13744681 ]
angeline shen commented on CLOUDSTACK-4199: ------------------------------------------- Testing RVR using 8/19/13 latest build #465 MS 10.223.195.52 host 10.223.50.3 XS 6.2 1. advance zone with XEN 6.2 cluster 2. create network offering with RVR enabled id: 17 name: rvrallow uuid: bc34384a-7704-4786-87cc-e5cc24f32dcf unique_name: rvrallow display_text: rvrallow nw_rate: NULL mc_rate: 10 traffic_type: Guest tags: NULL system_only: 0 specify_vlan: 0 service_offering_id: NULL conserve_mode: 0 created: 2013-08-19 22:34:41 removed: NULL default: 0 availability: Optional dedicated_lb_service: 1 shared_source_nat_service: 0 sort_key: 0 redundant_router_service: 1 state: Enabled guest_type: Isolated elastic_ip_service: 0 eip_associate_public_ip: 0 elastic_lb_service: 0 specify_ip_ranges: 0 inline: 0 is_persistent: 0 internal_lb: 0 public_lb: 1 egress_default_policy: 1 concurrent_connections: NULL 17 rows in set (0.00 sec) 3. As admin deploy VM using above nw offering rvrallow: id: 10 name: z1rvrallowadminV40 uuid: 16eb63bd-53fc-45a9-8d28-c77c82f86dc0 instance_name: i-2-10-VM state: Running vm_template_id: 5 guest_os_id: 12 private_mac_address: 02:00:63:30:00:06 private_ip_address: 10.1.1.23 pod_id: 1 data_center_id: 1 host_id: 1 last_host_id: 1 proxy_id: 2 proxy_assign_time: 2013-08-20 00:45:33 vnc_password: YGzUsIS8Z+37W9Udm2+S4zvsV07NT9bP ha_enabled: 1 limit_cpu_use: 0 update_count: 3 update_time: 2013-08-19 22:46:09 created: 2013-08-19 22:42:40 removed: NULL type: User vm_type: User account_id: 2 domain_id: 1 service_offering_id: 12 reservation_id: 3af80789-c7ea-47e8-a79b-8c1cc6b284de hypervisor_type: XenServer disk_offering_id: NULL cpu: NULL ram: NULL owner: 2 speed: 100 host_name: z1rvrallowadminV40 display_name: z1rvrallowadminV40 desired_state: NULL dynamically_scalable: 1 display_vm: 1 4. The above steps deployed RVR routers without any issues: id: 11 <=== MASTER name: r-11-VM uuid: 67bd2d37-cc5a-45ce-8cf7-74b06e1d9c8f instance_name: r-11-VM state: Running vm_template_id: 1 guest_os_id: 183 private_mac_address: 0e:00:a9:fe:02:c4 private_ip_address: 169.254.2.196 pod_id: 1 data_center_id: 1 host_id: 1 last_host_id: 1 proxy_id: NULL proxy_assign_time: NULL vnc_password: trOmaP4q+opdndMZM6Wayg4svTXwTje2WCbEWXYp8Ic= ha_enabled: 0 limit_cpu_use: 0 update_count: 29 update_time: 2013-08-20 02:25:07 created: 2013-08-19 22:42:41 removed: NULL type: DomainRouter vm_type: DomainRouter account_id: 2 domain_id: 1 service_offering_id: 7 reservation_id: c881947a-1737-410e-9d92-3b63268e6c53 hypervisor_type: XenServer disk_offering_id: NULL cpu: NULL ram: NULL owner: NULL speed: NULL host_name: NULL display_name: NULL desired_state: NULL dynamically_scalable: 0 display_vm: 1 *************************** 12. row *************************** id: 12 <=== BACKUP name: r-12-VM uuid: be11be35-d844-44fd-ae54-4656d9433855 instance_name: r-12-VM state: Running vm_template_id: 1 guest_os_id: 183 private_mac_address: 0e:00:a9:fe:02:12 private_ip_address: 169.254.2.18 pod_id: 1 data_center_id: 1 host_id: 1 last_host_id: 1 proxy_id: NULL proxy_assign_time: NULL vnc_password: DRNlYWFm2jI1JxaQSm6r2EHvPxT+JMCu5FDxW9QwUhQ= ha_enabled: 0 limit_cpu_use: 0 update_count: 11 update_time: 2013-08-19 22:45:57 created: 2013-08-19 22:42:41 removed: NULL type: DomainRouter vm_type: DomainRouter account_id: 2 domain_id: 1 service_offering_id: 7 reservation_id: 37d39fd6-335b-4cda-a925-cb9f45b3305f hypervisor_type: XenServer disk_offering_id: NULL cpu: NULL ram: NULL owner: NULL speed: NULL host_name: NULL display_name: NULL desired_state: NULL dynamically_scalable: 0 display_vm: 1 5. Stop the MASTER VR from CloudStack Observations: i. MASTER router went into stopped state successfully but, BACKUP router stuck in "FAULT" state forever. Here is the snippet of keepalived.log for FAULT router: root@r-12-VM:~# cat /ramdisk/rrouter/keepalived.log To backup called Disable public ip 0 Password server is not running Stopping DNS forwarder and DHCP server: dnsmasq(not running) ... (warning). cache internal: current active connections: 0 connections created: 0 failed: 0 connections updated: 0 failed: 0 connections destroyed: 0 failed: 0 cache external: current active connections: 0 connections created: 0 failed: 0 connections updated: 0 failed: 0 connections destroyed: 0 failed: 0 traffic processed: 0 Bytes 0 Pckts multicast traffic (active device=eth0): 16 Bytes sent 24 Bytes recv 1 Pckts sent 2 Pckts recv 0 Error send 0 Error recv message tracking: 0 Malformed msgs 0 Lost msgs Conntrackd switch to backup done Switch conntrackd mode backup 0 Status: BACKUP To master called ifdown: interface eth2 not configured RTNETLINK answers: File exists Failed to bring up eth2. RTNETLINK answers: No such process Enable public ip returned 2 Fail to enable public ip! Password server is not running Stopping DNS forwarder and DHCP server: dnsmasq(not running) ... (warning). Stopping keepalived: keepalived. Stopping conntrackd. Status: FAULT (RTNETLINK answers: No such process) root@r-12-VM:~# 6. MASTER router log: root@r-11-VM:~# cat /ramdisk/rrouter/keepalived.log To backup called Disable public ip 0 Password server is not running Stopping DNS forwarder and DHCP server: dnsmasq. cache internal: current active connections: 0 connections created: 0 failed: 0 connections updated: 0 failed: 0 connections destroyed: 0 failed: 0 cache external: current active connections: 0 connections created: 0 failed: 0 connections updated: 0 failed: 0 connections destroyed: 0 failed: 0 traffic processed: 0 Bytes 0 Pckts multicast traffic (active device=eth0): 40 Bytes sent 0 Bytes recv 5 Pckts sent 0 Pckts recv 0 Error send 0 Error recv message tracking: 0 Malformed msgs 0 Lost msgs Conntrackd switch to backup done Switch conntrackd mode backup 0 Status: BACKUP To master called ifdown: interface eth2 not configured Password server is not running Removed cloud-passwd-srvr iptables rules Added cloud-passwd-srvr iptables rules Restarting DNS forwarder and DHCP server: dnsmasq10.1.1.236/24 10.1.1.1/24 . Enable public ip returned 0 Conntrackd switch to primary done Switch conntrackd mode primary returned 0 ARPING 10.223.122.18 from 10.223.122.18 eth2 Sent 1 probes (1 broadcast(s)) Received 0 response(s) ARPING 10.223.122.18 from 10.223.122.18 eth2 Sent 1 probes (1 broadcast(s)) Received 0 response(s) Status: MASTER root@r-11-VM:~# 7. MASTER * ifconfig output * ifconfig -a output * /ramdisk/rrouter/keepalived.log * checkrouter.sh output d. BACKUP (before and after failover) * ifconfig output * ifconfig -a output * /ramdisk/rrouter/keepalived.log * checkrouter.sh output root@r-11-VM:~# ifconfig eth0 Link encap:Ethernet HWaddr 02:00:67:29:00:02 inet addr:10.1.1.236 Bcast:10.1.1.255 Mask:255.255.255.0 inet6 addr: fe80::67ff:fe29:2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1614 errors:0 dropped:0 overruns:0 frame:0 TX packets:11505 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:95725 (93.4 KiB) TX bytes:627963 (613.2 KiB) Interrupt:22 eth1 Link encap:Ethernet HWaddr 0e:00:a9:fe:02:c4 inet addr:169.254.2.196 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::c00:a9ff:fefe:2c4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8826 errors:0 dropped:0 overruns:0 frame:0 TX packets:8048 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1239708 (1.1 MiB) TX bytes:1493980 (1.4 MiB) Interrupt:23 eth2 Link encap:Ethernet HWaddr 06:23:60:00:00:18 inet addr:10.223.122.18 Bcast:10.223.122.63 Mask:255.255.255.192 inet6 addr: fe80::423:60ff:fe00:18/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:127 errors:0 dropped:0 overruns:0 frame:0 TX packets:218 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13423 (13.1 KiB) TX bytes:16317 (15.9 KiB) Interrupt:24 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:12 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2470 (2.4 KiB) TX bytes:2470 (2.4 KiB) root@r-11-VM:~# ifconfig -a eth0 Link encap:Ethernet HWaddr 02:00:67:29:00:02 inet addr:10.1.1.236 Bcast:10.1.1.255 Mask:255.255.255.0 inet6 addr: fe80::67ff:fe29:2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1648 errors:0 dropped:0 overruns:0 frame:0 TX packets:11720 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:97847 (95.5 KiB) TX bytes:639723 (624.7 KiB) Interrupt:22 eth1 Link encap:Ethernet HWaddr 0e:00:a9:fe:02:c4 inet addr:169.254.2.196 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::c00:a9ff:fefe:2c4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9084 errors:0 dropped:0 overruns:0 frame:0 TX packets:8276 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1266636 (1.2 MiB) TX bytes:1530404 (1.4 MiB) Interrupt:23 eth2 Link encap:Ethernet HWaddr 06:23:60:00:00:18 inet addr:10.223.122.18 Bcast:10.223.122.63 Mask:255.255.255.192 inet6 addr: fe80::423:60ff:fe00:18/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:128 errors:0 dropped:0 overruns:0 frame:0 TX packets:218 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13451 (13.1 KiB) TX bytes:16317 (15.9 KiB) Interrupt:24 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:12 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2470 (2.4 KiB) TX bytes:2470 (2.4 KiB) root@r-11-VM:~# checkrouter.sh Status: MASTER&Bumped: NO 8. BACKUP: root@r-12-VM:~# ifconfig eth0 Link encap:Ethernet HWaddr 02:00:50:bd:00:03 inet addr:10.1.1.75 Bcast:10.1.1.255 Mask:255.255.255.0 inet6 addr: fe80::50ff:febd:3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:31164 errors:0 dropped:0 overruns:0 frame:0 TX packets:8935 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1254816 (1.1 MiB) TX bytes:515974 (503.8 KiB) Interrupt:24 eth1 Link encap:Ethernet HWaddr 0e:00:a9:fe:02:12 inet addr:169.254.2.18 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::c00:a9ff:fefe:212/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:29972 errors:0 dropped:0 overruns:0 frame:0 TX packets:27342 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4248712 (4.0 MiB) TX bytes:5123688 (4.8 MiB) Interrupt:25 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:214 (214.0 B) TX bytes:214 (214.0 B) root@r-12-VM:~# ifconfig -a eth0 Link encap:Ethernet HWaddr 02:00:50:bd:00:03 inet addr:10.1.1.75 Bcast:10.1.1.255 Mask:255.255.255.0 inet6 addr: fe80::50ff:febd:3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:31197 errors:0 dropped:0 overruns:0 frame:0 TX packets:8935 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1256068 (1.1 MiB) TX bytes:515974 (503.8 KiB) Interrupt:24 eth1 Link encap:Ethernet HWaddr 0e:00:a9:fe:02:12 inet addr:169.254.2.18 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::c00:a9ff:fefe:212/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:30080 errors:0 dropped:0 overruns:0 frame:0 TX packets:27442 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4258400 (4.0 MiB) TX bytes:5139200 (4.9 MiB) Interrupt:25 eth2 Link encap:Ethernet HWaddr 06:23:60:00:00:18 inet addr:10.223.122.18 Bcast:10.223.122.63 Mask:255.255.255.192 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Interrupt:26 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:214 (214.0 B) TX bytes:214 (214.0 B) root@r-12-VM:~# checkrouter.sh Status: FAULT (RTNETLINK answers: No such process)&Bumped: NO > Redundant Virtual Router - no failover occur > -------------------------------------------- > > Key: CLOUDSTACK-4199 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4199 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Environment: MS ACS 4.2 campo internal build 341 > host XS 6.2 > Reporter: angeline shen > Assignee: Sheng Yang > Fix For: 4.2.0 > > Attachments: FAULT_logs.tgz, logs.tgz, management-server.log.gz, > MASTER_logs.tgz, Screenshot-CloudPlatform™ - Mozilla Firefox-3.png, > Screenshot-CloudPlatform™ - Mozilla Firefox-4.png > > > 1. create network offering 'egallowrvrnw1' with egress firewall policy : > allow , redundant router > advance zone. create network of this offering. create guest VMs > Verify ssh to VMs. VMs can ping other VMs in this network & reach > internet > 2. RVR MASTER r-37-VM > RVR BACKUP r-38-VM > stop r-37-VM > Result: r-37-VM state becomes UNKNOWN > r-38-VM state becomes FAULT > no failover occur > Cannot ssh to existing VMs > 3. start r-37-VM. > Result: r-37-VM state becomes MASTER > r-38-VM state remains FAULT > VMs can reach other VMs in same network. > VMs cannot reach internet > 4. stop r-37-VM > r-37-VM state becomes UNKNOWN > r-38-VM state becomes FAULT > no failover occur > Cannot ssh to existing VMs > r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 > networks to update RvR status. > 2013-08-08 19:22:44,763 INFO > [network.router.VirtualNetworkApplianceManagerImpl] > (RedundantRouterStatusMonitor-6:null) Redundant virtual router (name: > r-37-VM, id: 37) just switch from MASTER to UNKNOWN > 2013-08-08 19:22:44,768 DEBUG [agent.transport.Request] > (RedundantRouterStatusMonitor-6:null) Seq 1-2062888873: Sending { Cmd , > MgmtId: 7343890761426, via: 1, Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.CheckRouterCommand":{"a > ccessDetails":{"router.ip":"169.254.3.245","router.name":"r-38-VM"},"wait":30}}] > } > 2013-08-08 19:22:44,769 DEBUG [agent.transport.Request] > (RedundantRouterStatusMonitor-6:null) Seq 1-2062888873: Executing: { Cmd , > MgmtId: 7343890761426, via: 1, Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.CheckRouterCommand": > 2013-08-08 19:22:45,116 INFO > [network.router.VirtualNetworkApplianceManagerImpl] > (RedundantRouterStatusMonitor-6:null) Redundant virtual router (name: > r-38-VM, id: 38) just switch from BACKUP to FAULT > 2013-08-08 19:22:45,344 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-270:null) Seq 1-2062888874: Response Received: > 2013-08-08 19:22:45,345 DEBUG [agent.transport.Request] > (DirectAgent-270:null) Seq 1-2062888874: Processing: { Ans: , MgmtId: > 7343890761426, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"FAULT"," > isBumped":false,"result":true,"details":"Status: FAULT (RTNETLINK answers: No > such process)&Bumped: NO","wait":0}}] } > 2013-08-08 19:22:45,345 DEBUG [agent.transport.Request] > (RedundantRouterStatusMonitor-6:null) Seq 1-2062888874: Received: { Ans: , > MgmtId: 7343890761426, via: 1, Ver: v1, Flags: 10, { CheckRouterAnswer } } > 2013-08-08 19:22:45,345 DEBUG [agent.manager.AgentManagerImpl] > (RedundantRouterStatusMonitor-6:null) Details from executing class > com.cloud.agent.api.CheckRouterCommand: Status: FAULT (RTNETLINK answers: No > such process)&Bumped: N > O > 2013-08-08 19:22:45,349 INFO > [network.router.VirtualNetworkApplianceManagerImpl] > (RedundantRouterStatusMonitor-6:null) Redundant virtual router (name: > r-38-VM, id: 38) just switch from BACKUP to FAULT > 2013-08-08 19:22:46,781 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-13:null) Ping from 2 > 2013-08-08 19:22:47,125 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-12:null) Ping from 3 > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira