[
https://issues.apache.org/jira/browse/CLOUDSTACK-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016945#comment-14016945
]
ASF subversion and git services commented on CLOUDSTACK-5505:
-------------------------------------------------------------
Commit 5e80e5d33d9a295b91cdba9377f52d9d963d802a in cloudstack's branch
refs/heads/4.4-forward from Edison Su
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=5e80e5d ]
CLOUDSTACK-5505: if vpc public network with snat enabled, then will triger this
issue
> [Automation] Private gateway not getting programmed in VPC router
> ------------------------------------------------------------------
>
> Key: CLOUDSTACK-5505
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5505
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Network Controller
> Affects Versions: 4.3.0
> Environment: KVM
> Branch : 4.3
> Reporter: Rayees Namathponnan
> Assignee: Jayapal Reddy
> Priority: Critical
> Fix For: 4.4.0
>
> Attachments: CLOUDSTACK-5505.rar
>
>
> This issue found as part of regression automation
> integration.component.test_vpc_vms_deployment.TestVMDeployVPC.test_07_delete_network_with_rules
>
> Test cases performing below steps
> 1726 # Validate the following
> 1727 # 1. Create a VPC with cidr - 10.1.1.1/16
> 1728 # 2. Add network1(10.1.1.1/24) and network2(10.1.2.1/24) to this
> VPC.
> 1729 # 3. Deploy vm1 and vm2 in network1 and vm3 and vm4 in network2.
> 1730 # 4. Create a PF /Static Nat/LB rule for vms in network1.
> 1731 # 5. Create a PF /Static Nat/LB rule for vms in network2.
> 1732 # 6. Create ingress network ACL for allowing all the above rules
> from
> 1733 # public ip range on network1 and network2.
> 1734 # 7. Create egress network ACL for network1 and network2 to
> access
> 1735 # google.com.
> 1736 # 8. Create a private gateway for this VPC and add a static
> route to
> 1737 # this gateway
> 1738 # 9. Create a VPN gateway for this VPC and add a static route to
> this
> 1739 # gateway.
> 1740 # 10. Make sure that all the PF,LB, Static NAT rules work as
> expected
> 1741 # 11. Make sure that we are able to access google from all user
> Vms
> 1742 # 12. Make sure that the newly added private gateway's and VPN
> 1743 # gateway's static routes work as expected.
> 1744 # Steps:
> 1745 # 1. Delete the 1st network.
> 1746 # 2. Delete account
> 1747 # Validations:
> 1748 # 1. As part of network deletion all the resources attached with
> 1749 # network should get deleted. All other VMs and rules shall
> work as
> 1750 # expected
> 1751 # 2. All the resources associated with account should be deleted
> Steps to reproduce
> Step 1 : Create advanced zone in KVM
> Step 2 : Create VPC
> Step 3 : create 2 network inside
> Step 4 : Create Private Gateway
> Step 5 : Add static route
> Result
> Failed to add static route with error "Failed to create static route"
> MS log (look lob Job-Executor-102:ctx-12de6228)
> -------------------------------------------------------------------
> 2013-12-13 15:23:12,952 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-7:ctx-0c9be854) ===START=== 10.214.5.33 -- GET
> command=createStaticRoute&response=json&sessionkey=RP7BqoAP0Roa1gkg68z4YhvFAO4%3D&g
> atewayid=528f0ba2-bd3d-4e76-9d45-d567cbd47d67&cidr=10.2.3.0%2F24&_=1386976993256
> 2013-12-13 15:23:12,964 DEBUG [c.c.n.v.VpcManagerImpl]
> (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) Adding static route
> StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4]
> 2013-12-13 15:23:13,049 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-102:ctx-12de6228) Add job-2928 into job monitoring
> 2013-12-13 15:23:13,049 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-102:ctx-12de6228) Executing AsyncJobVO {id:2928, userId: 2,
> accountId: 2, instanceType: StaticRoute, instanceId: 4, cm
> d: org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd, cmdInfo:
> {"id":"4","response":"json","sessionkey":"RP7BqoAP0Roa1gkg68z4YhvFAO4\u003d","cmdEventType":"STATIC.ROUTE.CREATE","ctxU
> serId":"2","gatewayid":"528f0ba2-bd3d-4e76-9d45-d567cbd47d67","httpmethod":"GET","_":"1386976993256","ctxAccountId":"2","ctxStartEventId":"14430","cidr":"10.2.3.0/24"},
> cmdVersion: 0, status: IN_P
> ROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: null}
> 2013-12-13 15:23:13,053 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) submit async job-2928, details:
> AsyncJobVO {id:2928, userId: 2, accountId: 2, instanceTy
> pe: StaticRoute, instanceId: 4, cmd:
> org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd, cmdInfo:
> {"id":"4","response":"json","sessionkey":"RP7BqoAP0Roa1gkg68z4YhvFAO4\u003d","cmdEven
> tType":"STATIC.ROUTE.CREATE","ctxUserId":"2","gatewayid":"528f0ba2-bd3d-4e76-9d45-d567cbd47d67","httpmethod":"GET","_":"1386976993256","ctxAccountId":"2","ctxStartEventId":"14430","cidr":"10.2.3.0
> /24"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 29066118877352, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2013-12-13 15:23:13,055 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) ===END=== 10.214.5.33 -- GET
> command=createStaticRoute&response=json&sessionkey=RP7BqoAP0Roa1gkg68z4Y
> hvFAO4%3D&gatewayid=528f0ba2-bd3d-4e76-9d45-d567cbd47d67&cidr=10.2.3.0%2F24&_=1386976993256
> 2013-12-13 15:23:13,060 DEBUG [c.c.n.v.VpcManagerImpl]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applying static routes for vpc
> [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,070 DEBUG [c.c.a.t.Request]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592525: Sending { Cmd
> , MgmtId: 29066118877352, via: 2(Rack2Host12.lab.vmops.com), Ver: v1,
> Flags: 100001,
> [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":4,"uuid":"7c159911-d38a-4dbf-a98d-708b099c7389","targetCidr":"10.2.3.0/24","accountId":2,"domainId":1,"g
> atewayId":4,"state":"Add","vpcId":77,"vlanTag":"2364","gateway":"10.2.3.1","netmask":"255.255.255.0","ipAddress":"10.2.3.2"}],"accessDetails":{"zone.network.type":"Advanced","router.ip":"169.254.2
> .108","router.name":"r-706-QA"},"wait":0}}] }
> 2013-12-13 15:23:13,308 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-12:null) Seq 2-1828592525: Processing: { Ans: ,
> MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, [{"com.cloud.agent.api.rou
> ting.SetStaticRouteAnswer":{"results":["Failed"],"result":false,"wait":0}}] }
> 2013-12-13 15:23:13,308 DEBUG [c.c.a.t.Request]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592525: Received: {
> Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, { SetStaticRou
> teAnswer } }
> 2013-12-13 15:23:13,309 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-14:ctx-75d4f4a5) ===START=== 10.223.240.194 -- GET
> signature=m9opIJ%2BNeEGTeQ0NEl5GZnZtitU%3D&apiKey=LepyvCSNC55jvLUEoHU3GP9oI4QiT
> vgt9dsCdALVIf3NxlNBq7dmcHdCb_6NenlyV3QwRmEn5DnGp__U-5UwXA&command=queryAsyncJobResult&response=json&jobid=08d6fc64-19d4-433b-bcd2-20d487e43dd5
> 2013-12-13 15:23:13,314 DEBUG [c.c.n.v.VpcManagerImpl]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Revoking static route
> StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4]
> 2013-12-13 15:23:13,319 DEBUG [c.c.n.v.VpcManagerImpl]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Marked static route
> StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4] with state R
> evoke
> 2013-12-13 15:23:13,322 DEBUG [c.c.n.v.VpcManagerImpl]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applying static routes for vpc
> [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,331 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-14:ctx-75d4f4a5 ctx-e2e99bdf ctx-a1b9efd2) ===END===
> 10.223.240.194 -- GET signature=m9opIJ%2BNeEGTeQ0NEl5GZnZtitU%3D&apiKey=Lepyv
> CSNC55jvLUEoHU3GP9oI4QiTvgt9dsCdALVIf3NxlNBq7dmcHdCb_6NenlyV3QwRmEn5DnGp__U-5UwXA&command=queryAsyncJobResult&response=json&jobid=08d6fc64-19d4-433b-bcd2-20d487e43dd5
> 2013-12-13 15:23:13,340 DEBUG [c.c.a.t.Request]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592526: Sending { Cmd
> , MgmtId: 29066118877352, via: 2(Rack2Host12.lab.vmops.com), Ver: v1,
> Flags: 100001,
> [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":4,"uuid":"7c159911-d38a-4dbf-a98d-708b099c7389","targetCidr":"10.2.3.0/24","accountId":2,"domainId":1,"gatewayId":4,"state":"Revoke","vpcId":77,"vlanTag":"2364","gateway":"10.2.3.1","netmask":"255.255.255.0","ipAddress":"10.2.3.2"}],"accessDetails":{"zone.network.type":"Advanced","router.ip":"169.254.2.108","router.name":"r-706-QA"},"wait":0}}]
> }
> 2013-12-13 15:23:13,563 DEBUG [c.c.a.t.Request]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592526: Received: {
> Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, {
> SetStaticRouteAnswer } }
> 2013-12-13 15:23:13,563 DEBUG [c.c.n.e.VpcVirtualRouterElement]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applied static routes on vpc
> [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,563 DEBUG [c.c.n.v.VpcManagerImpl]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applied static routes for vpc
> [VPC [77-RayeesVPC]
> 2013-12-13 15:23:13,567 DEBUG [c.c.n.NetworkModelImpl]
> (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Service SecurityGroup is not
> supported in the network id=572
> 2013-12-13 15:23:13,574 DEBUG [c.c.n.v.VpcManagerImpl]
> (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Removed route
> StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4] from the DB
> 2013-12-13 15:23:13,578 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-102:ctx-12de6228) Complete async job-2928, jobStatus: FAILED,
> resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create static route"}
> 2013-12-13 15:23:13,584 DEBUG [c.c.n.NetworkModelImpl]
> (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Service SecurityGroup is not
> supported in the network id=572
> 2013-12-13 15:23:13,589 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Applying dhcp entry in network
> Ntwk[572|Guest|156]
> 2013-12-13 15:23:13,597 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-102:ctx-12de6228) Done executing
> org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd for job-2928
> 2013-12-13 15:23:13,601 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-102:ctx-12de6228) Remove job-2928 from job monitoring
> 2013-12-13 15:23:13,603 DEBUG [c.c.a.t.Request] (Job-Executor-15:ctx-f757750b
> ctx-4a7c2e5e) Seq 1-1935872252: Sending { Cmd , MgmtId: 29066118877352, via:
> 1(Rack2Host11.lab.vmops.com), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:42:c8:00:01","vmIpAddress":"10.1.1.122","vmName":"QA-2ac35f13-8ced-4c76-b983-5bdea778e276","defaultRouter":"10.1.1.1","duid":"00:03:00:01:02:00:42:c8:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-732-QA","router.ip":"169.254.1.101"},"wait":0}}]
> }
> Router Log
> -------------
> if you look the message.log in side the VPC router, you can see Private
> Gateway itself not programmed (attaching all log, please check router log
> r-706-QA)
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160401] pci 0000:00:0b.0: BAR 6:
> assigned [mem 0x08040000-0x0804ffff pref]
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160406] pci 0000:00:0b.0: BAR 1:
> assigned [mem 0x08013000-0x08013fff]
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160443] pci 0000:00:0b.0: BAR 0:
> assigned [io 0x1060-0x107f]
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160483] pci 0000:00:00.0: no hotplug
> settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160486] pci 0000:00:00.0: using
> default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160536] pci 0000:00:01.0: no hotplug
> settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160539] pci 0000:00:01.0: using
> default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160584] ata_piix 0000:00:01.1: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160587] ata_piix 0000:00:01.1: using
> default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160637] uhci_hcd 0000:00:01.2: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160641] uhci_hcd 0000:00:01.2: using
> default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160818] piix4_smbus 0000:00:01.3: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160822] piix4_smbus 0000:00:01.3:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160872] pci 0000:00:02.0: no hotplug
> settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.160875] pci 0000:00:02.0: using
> default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163389] virtio-pci 0000:00:03.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163393] virtio-pci 0000:00:03.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163430] virtio-pci 0000:00:04.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163433] virtio-pci 0000:00:04.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163488] virtio-pci 0000:00:05.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163491] virtio-pci 0000:00:05.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163532] virtio-pci 0000:00:06.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163535] virtio-pci 0000:00:06.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163581] virtio-pci 0000:00:07.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163584] virtio-pci 0000:00:07.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163640] virtio-pci 0000:00:08.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163644] virtio-pci 0000:00:08.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163795] virtio-pci 0000:00:09.0: no
> hotplug settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163798] virtio-pci 0000:00:09.0:
> using default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163855] pci 0000:00:0b.0: no hotplug
> settings from platform
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.163858] pci 0000:00:0b.0: using
> default PCI settings
> Dec 13 23:45:39 r-706-QA kernel: [ 2365.166368] virtio-pci 0000:00:0b.0:
> enabling device (0000 -> 0003)
> Dec 13 23:45:39 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 0 seconds
> Dec 13 23:45:40 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 1 seconds
> Dec 13 23:45:41 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 2 seconds
> Dec 13 23:45:42 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 3 seconds
> Dec 13 23:45:43 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 4 seconds
> Dec 13 23:45:44 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 5 seconds
> Dec 13 23:45:45 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 6 seconds
> Dec 13 23:45:46 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 7 seconds
> Dec 13 23:45:47 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 8 seconds
> Dec 13 23:45:48 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 9 seconds
> Dec 13 23:45:49 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 10 seconds
> Dec 13 23:45:50 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 11 seconds
> Dec 13 23:45:51 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 12 seconds
> Dec 13 23:45:52 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 13 seconds
> Dec 13 23:45:53 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 14 seconds
> Dec 13 23:45:54 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 15 seconds
> Dec 13 23:45:55 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull
> to appear, 16 seconds
> Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:interface ethnull never
> appeared
> Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:Adding ip 10.2.3.2 on
> interface ethnull
> Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:Add routing 10.2.3.2 on
> interface ethnull
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: enter apply acl rules
> on private gateway interface : eth4, inbound::all:0:0:0.0.0.0/0
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: exit apply acl rules
> for private gw interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: successful in applying
> acl rules on private gateway interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: enter apply acl rules
> on private gateway interface : eth4, inbound::all:0:0:0.0.0.0/0
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: exit apply acl rules
> for private gw interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: successful in applying
> acl rules on private gateway interface : eth4
> Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: deleting backup on
> private gateway interface : eth4
--
This message was sent by Atlassian JIRA
(v6.2#6252)
