Re: [Openstack-operators] [neutron] Packet loss with DVR and IPv6
Tomas Vondra writes: > The setup has 3 network nodes and 1 compute node currently hosting a virtual > network (GRE based). DVR is enabled. I have just added IPv6 to this network > and to the external network (VLAN based). The virtual network is set to SLAAC. > > However, the link-local router address and associated MAC address is the > same in all 4 qr namespaces. About 16% packets get lost in randomly occuring > bursts. Openvswitch forwarding tables are flapping and I think that the > packet loss occurs at the moment when all 4 switches learn the MAC address > through a GRE tunnel simultaneously. I'm convinced it is a bug that was probably not fixed yet. Filing bug report. https://bugs.launchpad.net/neutron/+bug/1596473 Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] [neutron] Packet loss with DVR and IPv6
Dear list, I've stumbled upon a weird condition in Neutron and couldn't find a bug filed for it. So even if it is happening with the Kilo release, it could still be relevant. The setup has 3 network nodes and 1 compute node currently hosting a virtual network (GRE based). DVR is enabled. I have just added IPv6 to this network and to the external network (VLAN based). The virtual network is set to SLAAC. Now, all four mentioned nodes have spawned a radvd process and VMs are getting globally routable addresses. Traffic has been statically routed to the subnet so reachability is OK in both ways. However, the link-local router address and associated MAC address is the same in all 4 qr namespaces. About 16% packets get lost in randomly occuring bursts. Openvswitch forwarding tables are flapping and I think that the packet loss occurs at the moment when all 4 switches learn the MAC address through a GRE tunnel simultaneously. Another router address and the external gateway address resides in a snat namespace, which exists in only one copy. When I tell the VM to route through that, there is no packet loss. Is there any way to change the behavior to either get rid of the MAC address conflict of push a default route to the VMs through the stable gateway? Thanks, Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Problem in using ide disk_type and virtio network interface
shiva m writes: > > Hi, > I have an image with /dev/sda partitions (IDE disk type) and virtio network interfaces. I loaded my image on Openstack with followign command. > > glance image-create --name --disk-format qcow2 --container-format bare --property hw_disk_bus=ide --property hw_vif_model=virtio --is-public True --file > > > > I am able to launch the instane from this image but my network interfaces are not coming up. > > if I do lspci, I can see network interfaces are of type virtio > > > Can anyone please help in this regard. Am I missing anything. > > > Thanks, > Shiva > Hi Shiva, when you run `ip link`, does it show the interfaces? If no, the virtio driver is not loaded. If yes, your distro's network configuration is not set to turn them on and use DHCP. Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] How are folks providing GPU instance types?
Nordquist, Peter L writes: > You will also have to enable iommu on your hypervisors to have libvirt expose the capability to Nova for PCI > passthrough. I use Centos 7 and had to set 'iommu=pt intel_iommu=on' for my kernel parameters. Along with > this, you'll have to start using EFI for your VMs by installing OVMF on your Hypervisors and configuring > your images appropriately. I don't have a link handy for this but the gist is that Legacy bootloaders have a > much more complicated process to initialize the devices being passed to the VM where EFI is much easier. Hi! What I found out the hard way under the Xen hypervisor is that the GPU you are passing through must not be the primary GPU of the system. Otherwise, you get memory corruption as soon as something appears on the console. If not sooner :-). Test if your motherboards are capable of running on the integrated VGA even if some other graphics card is connected. Or blacklist it for the kernel. Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Compute node with different subnet
Ram Kumar writes: > > > We have three compute nodes with different subnets on the same vlan. When we create instance, we are getting an ip from first subnet but running on second node which is different subnet and can't ping gateway or traffic is not going out from that instance because node is running on different subnet. If instance is created on first node which is same subnet of that host, we don't see any issues. Please advice how to resolve this issue? > Thanks Ram You may want to look into project Romana.io or other pure-L3 networking plugins for OpenStack. If we are talking about default Neutron implementation, then your design seems to be wrong, you will need the same subnet over all compute nodes. You may have all subnets over all compute nodes, define corresponding provider networks in neuthon and schedule the VMs into them manually, though. Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] New networking solution for Cloud Native apps....
Clint Byrum writes: > > I also think you should share this on openstack-dev, as the developers > may also be aware of other efforts that may conflict with or complement > Romana. Your mission seems to be very close to that of Project Calico. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] DVR and public IP consumption
Carl Baldwin writes: > You're right, the IP in the fip namespace doesn't ever get written in > to any packets or used as an arp destination. It is currently > meaningless. That will change with BGP's capability to routed DVR > traffic in Mitaka because that IP will be used as a next hop. > However, it still doesn't need to be a public IP. The routed networks > work that I'm doing in Newton will allow us to eventually make these > private IPs instead of public so that public IPs are not wasted. > > I've given these things a lot of thought but haven't had time to > pursue any such thoughts yet except to implement routed networks as > groundwork. Here are a few old links [1][2] but they are really out > of date. I need to write another spec following the first routed > networks spec explaining how these things will work. > > Here is an etherpad [3] that I put together a couple of years ago > trying to compare different approaches to getting rid of centralized > SNAT too. We just never got any traction on any of these approaches. > Also, without the routed networks work in Newton, many of them are > difficult to accomplish. > > Let me know if anything resonates with you. We might be in a better > position to do some of this work when routed networks is under way. > For example, one thing that routed networks may allow is using private > IPs for the router's address. I think that was in one of the above > blueprints somewhere. Let me go write a new spec and post it. I'll > update this thread when I've got it up. > > Carl > > [1] https://review.openstack.org/#/c/174657/2/specs/liberty/eliminate-dvr-fip-ns.rst > [2] https://review.openstack.org/#/c/175517/1/specs/liberty/no-router-ip.rst > [3] https://etherpad.openstack.org/p/decentralized-snat > Hi Carl, sorry for the late reply, but these links of yours expanded to about 12 tabs in my browser, most with serveral pages of text. "Given lots of thought" may be an understatement. Both the specs sound very resonable to me. The second one is exactly what I was saying here before. (Evidently I was not the first.) Why was it not accepted? It seems quite easy to implement in contrast to full routed networks. The work on routed networks will be beneficial mainly for large deployments, whose needs exceed the capacity of a few L2 domains. Small public deployers are working on the scale of tens of boxes, but hundreds of tenants. Each tenant gets a virtual router, which eats an IP. I only have 1024 IPs from RIPE and will probably get no more. If most of the tenants are small and only use a one or two VMs, I'm wasting up to 50% addresses and it is severely limiting my growth potential. I do not really understand why routed networks would be a prerequisite to using private IPs for router interfaces. I'm aiming at the last point from the Etherpad - Carrier grade NAT. Do you think that I could use the "Allow setting a tenant router's external IP" function and disable any checks if the specified IP is in the network defined as external? I already have a private subnet on the same L2 segment, that is NATted by the datacenter routers. The API is admin-only, so it would not create a risk. I would pre-create a router for each tenant and everyone would be happy. Floating IPs are taken care of at the compute nodes in DVR. Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] New networking solution for Cloud Native apps....
Chris Marino writes: > > > Hello everyone, just wanted to let you know that today we opened up the repos for the new open source networking project we’ve been working on. It’s called Romana and the project site is romana.io. > Thought you would be interested because it enables multi-tenant networking without a virtual network overlay. It's targeted for use with applications that only need L3 networks so we’ve been able to eliminate and simplify many things to make the network faster, and easier to build and operate. > If you run these kind of Cloud Native apps on OpenStack (or even directly on bare metal with Docker or Kubernetes), we’d love to hear what you think. We’re still working on the container CNM/CNI integration. Any and all feedback is welcome. > The code is on Github at github.com/romana and you can see how it all works with a demo we’ve set up that lets you install and run OpenStack on EC2. > You can read about how Romana works on the project site, here. In summary, it extends the physical network hierarchy of a layer 3 routed access design from spine and leaf switches on to hosts, VMs and containers. > This enables a very simple and intuitive tenancy model: For every tenant (and each of their network segments) there is an actual physical network CIDR on each host, with all tenants sharing the host-specific address prefix. The advantage of this is that route aggregation makes route distribution unnecessary and collapses the number of iptables rules required for segment isolation. In addition, traffic policies, such as security rules, can easily be applied to those tenant or segment specific CIDRs across all hosts. > Any/all comments welcome. > Thanks > CM > Hi Chris! It's a very interesting project, I like the clean L3 design, without mixing OpenVSwitches and namespaces. But as for myself I'd lean more towards a full L2 implementetion. I for example liked the Dragonflow blog posts I have read. I like the way Provider Networks in Neutron allow me to connect physical boxes to VMs in OpenStack. My concerns are: 1) I'm a service provider. As long as I'm giving out my IP addresses to tenants, all is fine. But what if someone wants to bring their own? There should be a way out of the fixed addressing scheme in exceptional cases. 2) What about some legacy distributed apps, which scan the network to find all nodes? Think MPI. 3) What if I already have the 10.0.0.0/8 network used in the datacenter? (I think it will be, in most of them.) 4) What about floating IPs? Or will there be dynamic routing for internet access of individual instances? 5) Do you have a roadmap / feature matrix of what would you like to have and how it compares to other OpenStack solutions? 6) If you ever do another benchmark, focus more on throughput than latency. The difference between 1.5 ms and 0.24 ms is not interesting. The one between, say, 5 and 10 Gb/s would be. And please document the hardware setup ;-). Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] DVR and public IP consumption
Fox, Kevin M writes: > > Hi Tomas, > > The using external addresses per tenant router is a feature to a lot of sites, like ours. We want to know for > sure, at minimum, which tenant was responsible for bad activity on the external network. Having the > external address tied to a tenant router allows you to track bad activity back at least to the ip, then to the > tenant router. You won't be able to tell which vm's of the tenant performed the bad activity because of the > snat, but you at least have some to talk to about it, instead of your local security friends asking you to > unplug the whole cloud. > > Thanks, > Kevin Hi Kevin! Don't worry, I also had this in mind. We do traffic logging at the datacenter's firewall, so using a private IP per tenant router would still satisfy this requirement. Tomas ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] DVR and public IP consumption
raffic to the right router node. DVR as implemented is really just a local NAT gateway per physical compute node. The outside of your NAT needs to be publicly unique, > so it needs it's own address. Some SDN solutions can provide a truly distributed router model, because they globally know the inside state of the NAT environment, and can forward packets back to the internal source properly, regardless of which distributed > forwarder receives the incoming "external" packets. > > If the number of external addresses consumed is an issue, you may consider the dual gateway HA model instead of DVR. This uses classic multi-router models where one router takes on the task of forwading packets, and the other device just acts as a backup. > You do still have a software bottleneck at your router, unless you then also use one of the plugins that supports hardware L3 (last I checked, Juniper, Arista, Cisco, etc. all provide an L3 plugin that is HA capable), but you only burn 3 External addresses > for the router (and 3 internal network addresses per tenant side interface if that matters). > > Hope that clarifies a bit, > Robert > > > On Tue, Jan 26, 2016 at 4:14 AM, Carl Baldwin > ecbaldwin.net> wrote: > On Thu, Jan 14, 2016 at 2:45 AM, Tomas Vondra wrote: > > Hi! > > I have just deployed an OpenStack Kilo installation with DVR and expected > > that it will consume one Public IP per network node as per > > > http://assafmuller.com/2015/04/15/distributed-virtual-routing-floating-ips/, > > but it still eats one per virtual Router. > > What is the correct behavior?Regardless of DVR, a Neutron router burns one IP per virtual router > which it uses to SNAT traffic from instances that do not have floating > IPs. > When you use DVR, an additional IP is consumed for each compute host > running an L3 agent in DVR mode. There has been some discussion about > how this can be eliminated but no action has been taken to do this. > > Otherwise, it works as a DVR should according to documentation. There are > > router namespaces at both compute and network nodes, snat namespaces at the > > network nodes and fip namespaces at the compute nodes. Every router has a > > router_interface_distributed and a router_centralized_snat with private IPs, > > however the router_gateway has a public IP, which I would like to getr id of > > to increase density.I'm not sure if it is possible to avoid burning these IPs at this > time. Maybe someone else can chime in with more detail. > Carl ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] DVR and public IP consumption
Hi! I have just deployed an OpenStack Kilo installation with DVR and expected that it will consume one Public IP per network node as per http://assafmuller.com/2015/04/15/distributed-virtual-routing-floating-ips/, but it still eats one per virtual Router. What is the correct behavior? Otherwise, it works as a DVR should according to documentation. There are router namespaces at both compute and network nodes, snat namespaces at the network nodes and fip namespaces at the compute nodes. Every router has a router_interface_distributed and a router_centralized_snat with private IPs, however the router_gateway has a public IP, which I would like to getr id of to increase density. Thanks ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
