[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-576733004 @DaanHoogland I would like to wait for the PR of wei to be tested and merged. If the community accepts it then we can close the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-576162871 @weizhouapache Hi, sorry for the long delay. The last time I tested this I couldn't find any problems. The two setups I tested (KVM with Linux Bridge and KVM with OVS) looked very good This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-563124035 @weizhouapache I'll build and test it for OvS today. Anything you want me to test? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-560396224 @weizhouapache The "make redundant" worked fine for me. I also looked if private gateways go to the right interface and as far as I can tell this looks good. Can you tell me something about the issue with isolated networks so maybe I can replicate the error and give you some feedback? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-560294620 @weizhouapache Hi, no we don't use the ovs plugin. We have OvS in place for the hypervisor bridges. I'll resume testing today. I would like to test if maybe the make_redundant process offered for non-redundant VPC is working. Also I'll test if custom network offerings are posing problems This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-559018647 @ustcweizhou You are completely right. There was a ressource issue i failed to see. With that gone multiple VMs deploy just fine. I will start testing multiple public IPs and address ranges This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-558698138 @weizhouapache I tested multiple things that work fine (for redundant VPC offering): - [x] NAT is working (one NAT ip) - [x] Settings seem to go to both routers - [x] Redundancy heals when one router dies - [x] Failover of traffic works when one master dies - [x] Adding multiple Guest Networks works For things that do not work I have found: At the moment it's only possible to deploy one VM in a GuestNet (when using redundant VPC with OvS). For the second VM an error occurs. If one deletes that one working VM it's possible to deploy again. But max 1 VM. Even if I try to deploy in another GuestNet (in the same VPC). Management log only shows: click to unfold ```INFO [o.a.c.a.c.u.v.DeployVMCmd] (API-Job-Executor-34:ctx-6242853f job-107 ctx-b55ef55c) (logid:852387c4) Unable to create a deployment for VM[User|i-4-18-VM] com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|i-4-18-VM]Scope=interface com.cloud.dc.DataCenter; id=1 at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:215) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:200) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4676) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4231) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4221) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy204.startVirtualMachine(Unknown Source) at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:588) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:583) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` Also I noticed another error in the management log that could be related: click to unfold ```ERROR [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-32:ctx-d8346e02 job-92 ctx-5a62bb49) (logid:c681419d) Failed to remove dhcp-dns
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-558551513 @weizhouapache The first test looks promising. The issue with the agent is gone. Redundant routers are deploying fine now. Also it looks like the VRRP IP is on the right interface. The routers are negotiating BACKUP and Master. I'll go on and test NAT and features. I'll report back when done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-558036516 @weizhouapache Ok, kindly give me a ping if there's something to test or help. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-557593215 @weizhouapache After a first look into the router I can say the APIPA is not the problem. There's something else wrong. I will have a look and come back to report This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-557580409 @weizhouapache Just writing a short feedback so you know where I'm at. At the moment it seems the redundant VPC does not deploy with your changes. After deploying cloudstack management isn't able to connect to the agent running on the first VPC router. (The second one never get's started because the first one fails.) My guess is that the APIPA address is the culprit. I will setup a systemvm with password access so I can look into the VPC router and give you more information on that. The non-redundant VPC deploys just fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-52466 @weizhouapache I'll setup a test environment to test the ovs part. It will take some days. I'll come back with feedback This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-554921846 @weizhouapache Leaving https://github.com/apache/cloudstack/pull/2304 in place sounds good. To me it looks like the more stable approach. Do you think the changes to get the isolated networks up to speed require some heavy lifting? If theres something to test I can do so. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-554407013 @weizhouapache Yes you are right: > the second function prepareNetworkElementCommand(final IpAssocVpcCommand cmd) does NOT work for normal setup (linux bridge), if we apply the change in #3179 (comment) That's the reason why I asked about applying the change in https://github.com/apache/cloudstack/issues/3179. We also thought about another bug beeing the root cause. A wrong MAC seems very plausible. I will help test the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-554361557 @weizhouapache The second function `prepareNetworkElementCommand(final IpAssocVpcCommand cmd)` does work for linux bridge and ovs at the moment I think. But it doen't work for the redundant vpc offering with ovs. Can you explain why you want to > (1) apply the change in #3179 (comment) Do you think it would be possible to write a mac address map based function for both (ovs and linux bridge)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong
interface
URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-553940664
@weizhouapache I found that the json the router receives really does not
have the "nic_dev_id" set at all. Therefor the script fails because the value
has "null" asigned then.
**redundant VPC**
`{"ip_address":[
{"public_ip":"172.20.20.12","source_nat":true,"add":true,"one_to_one_nat":false,"first_i_p":true,"gateway":"172.20.0.1","netmask":"255.255.0.0","vif_mac_address":"1e:00:d5:00:00:c2","new_nic":false,"nw_type":"Public"}],"type":"ips"}
**non redundant VPC**
{"ip_address":[
{"public_ip":"172.20.20.13","source_nat":true,"add":true,"one_to_one_nat":false,"first_i_p":true,"gateway":"172.20.0.1","netmask":"255.255.0.0","vif_mac_address":"1e:00:f4:00:00:c3",**"nic_dev_id":1**,"new_nic":false,"nw_type":"Public"}],"type":"ips"}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-553641686 @weizhouapache I tested now and hit a related error. In https://github.com/apache/cloudstack/blob/96d98de85cdce3ab9e33c6dda6ffa131d825cd85/systemvm/debian/opt/cloud/bin/cs/CsAddress.py#L261 it fails with: _self.dnum = hex(100 + int(dev[3:]))ValueError: invalid literal for int() with base 10: 'None'_ After that the deployment of the redundant vpc fails and the router is deleted (The second one does not even start to deploy). I'll try to step through code tomorrow and see why there is no device number. Also I will try to get my hands on to the generated json file. Do you see anything suspicious? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-552468180 @weizhouapache Thanks! Looks very promising. I will compile/test and come back to report if that fixes it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-552460679 Also we checked and found out, that the redundant VPC ist not using the https://github.com/apache/cloudstack/blob/9e6ee7c63e8b3073f9855511fe7cb004414b2dd4/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java#L1756 We think that this is very likely the root of this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-552457610 @weizhouapache The management logs are mostly uninteresting. But they also show that both VPC-router go into BACKUP state:   This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-551873157 @ustcweizhou It's very likely the issue is very old. My colleague mentioned it could be as old as 4.6 ... We are currently trying to find out more where the issue originates. It looks like the hint fmaximus gave is in parts true: In the json file the VPC-router receives, the dev_num and vif_mac address are wrong for the public ip. So whereever this gets matched the error happens. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-551762377 @ustcweizhou The single Router offering works like a charm in thus setup. We use that for hundreds of VMs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-551520649 Everytime we try to deploy redundant offering for VPCs. It only seems to happen with OvS bridges though. I tried to reproduce it with Linux Bridges but there it did work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-486180322 @ustcweizhou Are you using KVM with OVS? We are using advanced networking. Maybe we can pin this down by identifying the difference. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-484117369 The root cause seems to be that the interfaces (eth1, eth2, eth3) share the same mac address. I'm not really sure where this originates. Somehow when the redundant router is created and the nics are added all of them get the same mac. The "createRedundantAssociateIPCommands" seems to assign the ips to the wrong interface then. @ustcweizhou Can you give me a hint where in the code new vrouters get their nics? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface
DennisKonrad commented on issue #3179: Redundant VRouter guest network on wrong interface URL: https://github.com/apache/cloudstack/issues/3179#issuecomment-483998833 I found some more time looking into this. @ustcweizhou Do you also think the problem originates in https://github.com/apache/cloudstack/blob/52f68a273aaf0aa6746eaaee4a95a7bf722c53b0/core/src/main/java/com/cloud/agent/resource/virtualnetwork/VirtualRoutingResource.java#L401 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
