Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Hi Luke That sounds fantastic. As an NFV application developer I'm very pleased to see this contribution which looks to eliminate the key bottleneck hitting the performance of very high packet throughput apps on OpenStack. A couple of questions on features and implementation: 1. If I create a VM with say neutron and Open vSwitch then I get a TAP device + Linux bridge + veth device between the VM and the vSwitch, with the Linux bridge needed for implementing anti-spoofing iptables rules/ security group support. What will the stack look like with your NFV driver in place? Will your stack implement equivalent security functions, or will those functions not be available? 2. Are you planning to support live migration? cheers Calum Calum Loudon Director of Architecture Metaswitch Networks -Original Message- From: Luke Gorrie [mailto:l...@snabb.co] Sent: 10 January 2014 15:12 To: OpenStack Development Mailing List Cc: snabb-de...@googlegroups.com Subject: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron Howdy Stackers! We are developing a new open source Network Functions Virtualization driver for Neutron. I am writing to you now to ask for early advice that could help us to smoothly bring this work upstream into OpenStack Juno. The background is that we are open source developers working to satisfy the NFV requirements of large service provider networks including Deutsche Telekom's TeraStream project [1] [2]. We are developing a complete NFV stack for this purpose: from the DPDK-like traffic plane all the way up to the Neutron ML2 driver. We are developing against Havana, we attended the Icehouse summit and had a lot of great discussions in Hong Kong, and our ambition is to start bringing running code upstream into Juno. Our work is 100% open source and we want to work in the open with the wider OpenStack community. Currently we are in "heads-down hacking mode" on the core functionality, but it would be wonderful to connect with the upstream communities who we hope to be working with more in the future (that's you guys). More details on Github: https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv Thanks for reading! Cheers, -Luke [1] Ivan Pepelnjak on TeraStream: http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html [2] Peter Löthberg's presentation on TeraStream at RIPE 67: https://ripe67.ripe.net/archives/video/3/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Understood. You should be able to make that work but the issue is allocating your VM to some machine that has spare hardware - which is really what the patches are about, Nova manages allocations and Neutron manages using the hardware when appropriate. From past experience with the patch that was around back in the Essex timeframe, you can get this to work temporarily by rejecting the schedule in nova-compute when the machine is short of hardware and using a high schedule retry count, which will get you somewhere but is obviously a bit sucky in the long run. -- Ian. On 13 January 2014 18:44, Luke Gorrie wrote: > Howdy Ian! > > Thanks for the background on the Passthrough work. > > I reckon the best choice for us now is to use the traditional Neutron > APIs instead of Passthrough. I think they cover all of our use cases > as it stands now (many thanks to you for your earlier help with > working this out :)). The idea is to put the SR-IOV hardware to work > behind-the-scenes of a normal software switch. > > We will definitely check out the Passthrough when it's ready and see > if we should also support that somehow. > > > On 11 January 2014 01:04, Ian Wells wrote: > > Hey Luke, > > > > If you look at the passthrough proposals, the overview is that part of > the > > passthrough work is to ensure there's an PCI function available to > allocate > > to the VM, and part is to pass that function on to the Neutron plugin via > > conventional means. There's nothing that actually mandates that you > connect > > the SRIOV port using the passthrough mechanism, and we've been working on > > the assumption that we would be supporting the 'macvtap' method of > > attachment that Mellanox came up with some time ago. > > > > I think what we'll probably have is a set of standard attachments > (including > > passthrough) added to the Nova drivers - you'll see in the virtualisation > > drivers that Neutron already gets to tell Nova how to attach the port and > > can pass auxiliary information - and we will pass the PCI path and, > > optionally, other parameters to Neutron in the port-update that precedes > VIF > > plugging. That would leave you with the option of passing the path back > and > > requesting an actual passthrough or coming up with some other mechanism > of > > your own choosing (which may not involve changing Nova at all, if you're > > using your standard virtual plugging mechanism). > > > > -- > > Ian. > > > > > > On 10 January 2014 19:26, Luke Gorrie wrote: > >> > >> Hi Mike, > >> > >> On 10 January 2014 17:35, Michael Bright wrote: > >> > >> > Very pleased to see this initiative in the OpenStack/NFV space. > >> > >> Glad to hear it! > >> > >> > A dumb question - how do you see this related to the ongoing > >> > "[openstack-dev] [nova] [neutron] PCI pass-through network > support" > >> > > >> > discussion on this list? > >> > > >> > Do you see that work as one component within your proposed > architecture > >> > for > >> > example or an alternative implementation? > >> > >> Good question. I'd like to answer separately about the underlying > >> technology on the one hand and the OpenStack API on the other. > >> > >> The underlying technology of SR-IOV and IOMMU hardware capabilities > >> are the same in PCI pass-through and Snabb NFV. The difference is that > >> we introduce a very thin layer of software over the top that preserves > >> the basic zero-copy operation while adding a Virtio-net abstraction > >> towards the VM, packet filtering, tunneling, and policing (to start > >> off with). The design goal is to add quite a bit of functionality with > >> only a modest processing cost. > >> > >> The OpenStack API question is more open. How should we best map our > >> functionality onto Neutron APIs? This is something we need to thrash > >> out together with the community. Our current best guess - which surely > >> needs much revision, and is not based on the PCI pass-through > >> blueprint - is here: > >> > >> > https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv#neutron-configuration > >> > >> Cheers, > >> -Luke > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Howdy Ian! Thanks for the background on the Passthrough work. I reckon the best choice for us now is to use the traditional Neutron APIs instead of Passthrough. I think they cover all of our use cases as it stands now (many thanks to you for your earlier help with working this out :)). The idea is to put the SR-IOV hardware to work behind-the-scenes of a normal software switch. We will definitely check out the Passthrough when it's ready and see if we should also support that somehow. On 11 January 2014 01:04, Ian Wells wrote: > Hey Luke, > > If you look at the passthrough proposals, the overview is that part of the > passthrough work is to ensure there's an PCI function available to allocate > to the VM, and part is to pass that function on to the Neutron plugin via > conventional means. There's nothing that actually mandates that you connect > the SRIOV port using the passthrough mechanism, and we've been working on > the assumption that we would be supporting the 'macvtap' method of > attachment that Mellanox came up with some time ago. > > I think what we'll probably have is a set of standard attachments (including > passthrough) added to the Nova drivers - you'll see in the virtualisation > drivers that Neutron already gets to tell Nova how to attach the port and > can pass auxiliary information - and we will pass the PCI path and, > optionally, other parameters to Neutron in the port-update that precedes VIF > plugging. That would leave you with the option of passing the path back and > requesting an actual passthrough or coming up with some other mechanism of > your own choosing (which may not involve changing Nova at all, if you're > using your standard virtual plugging mechanism). > > -- > Ian. > > > On 10 January 2014 19:26, Luke Gorrie wrote: >> >> Hi Mike, >> >> On 10 January 2014 17:35, Michael Bright wrote: >> >> > Very pleased to see this initiative in the OpenStack/NFV space. >> >> Glad to hear it! >> >> > A dumb question - how do you see this related to the ongoing >> > "[openstack-dev] [nova] [neutron] PCI pass-through network support" >> > >> > discussion on this list? >> > >> > Do you see that work as one component within your proposed architecture >> > for >> > example or an alternative implementation? >> >> Good question. I'd like to answer separately about the underlying >> technology on the one hand and the OpenStack API on the other. >> >> The underlying technology of SR-IOV and IOMMU hardware capabilities >> are the same in PCI pass-through and Snabb NFV. The difference is that >> we introduce a very thin layer of software over the top that preserves >> the basic zero-copy operation while adding a Virtio-net abstraction >> towards the VM, packet filtering, tunneling, and policing (to start >> off with). The design goal is to add quite a bit of functionality with >> only a modest processing cost. >> >> The OpenStack API question is more open. How should we best map our >> functionality onto Neutron APIs? This is something we need to thrash >> out together with the community. Our current best guess - which surely >> needs much revision, and is not based on the PCI pass-through >> blueprint - is here: >> >> https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv#neutron-configuration >> >> Cheers, >> -Luke >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Hey Luke, If you look at the passthrough proposals, the overview is that part of the passthrough work is to ensure there's an PCI function available to allocate to the VM, and part is to pass that function on to the Neutron plugin via conventional means. There's nothing that actually mandates that you connect the SRIOV port using the passthrough mechanism, and we've been working on the assumption that we would be supporting the 'macvtap' method of attachment that Mellanox came up with some time ago. I think what we'll probably have is a set of standard attachments (including passthrough) added to the Nova drivers - you'll see in the virtualisation drivers that Neutron already gets to tell Nova how to attach the port and can pass auxiliary information - and we will pass the PCI path and, optionally, other parameters to Neutron in the port-update that precedes VIF plugging. That would leave you with the option of passing the path back and requesting an actual passthrough or coming up with some other mechanism of your own choosing (which may not involve changing Nova at all, if you're using your standard virtual plugging mechanism). -- Ian. On 10 January 2014 19:26, Luke Gorrie wrote: > Hi Mike, > > On 10 January 2014 17:35, Michael Bright wrote: > > > Very pleased to see this initiative in the OpenStack/NFV space. > > Glad to hear it! > > > A dumb question - how do you see this related to the ongoing > > "[openstack-dev] [nova] [neutron] PCI pass-through network support" > > > > discussion on this list? > > > > Do you see that work as one component within your proposed architecture > for > > example or an alternative implementation? > > Good question. I'd like to answer separately about the underlying > technology on the one hand and the OpenStack API on the other. > > The underlying technology of SR-IOV and IOMMU hardware capabilities > are the same in PCI pass-through and Snabb NFV. The difference is that > we introduce a very thin layer of software over the top that preserves > the basic zero-copy operation while adding a Virtio-net abstraction > towards the VM, packet filtering, tunneling, and policing (to start > off with). The design goal is to add quite a bit of functionality with > only a modest processing cost. > > The OpenStack API question is more open. How should we best map our > functionality onto Neutron APIs? This is something we need to thrash > out together with the community. Our current best guess - which surely > needs much revision, and is not based on the PCI pass-through > blueprint - is here: > > https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv#neutron-configuration > > Cheers, > -Luke > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Hi Mike, On 10 January 2014 17:35, Michael Bright wrote: > Very pleased to see this initiative in the OpenStack/NFV space. Glad to hear it! > A dumb question - how do you see this related to the ongoing > "[openstack-dev] [nova] [neutron] PCI pass-through network support" > > discussion on this list? > > Do you see that work as one component within your proposed architecture for > example or an alternative implementation? Good question. I'd like to answer separately about the underlying technology on the one hand and the OpenStack API on the other. The underlying technology of SR-IOV and IOMMU hardware capabilities are the same in PCI pass-through and Snabb NFV. The difference is that we introduce a very thin layer of software over the top that preserves the basic zero-copy operation while adding a Virtio-net abstraction towards the VM, packet filtering, tunneling, and policing (to start off with). The design goal is to add quite a bit of functionality with only a modest processing cost. The OpenStack API question is more open. How should we best map our functionality onto Neutron APIs? This is something we need to thrash out together with the community. Our current best guess - which surely needs much revision, and is not based on the PCI pass-through blueprint - is here: https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv#neutron-configuration Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Hi Luke, Very pleased to see this initiative in the OpenStack/NFV space. A dumb question - how do you see this related to the ongoing "[openstack-dev] [nova] [neutron] PCI pass-through network support" discussion on this list? Do you see that work as one component within your proposed architecture for example or an alternative implementation? Regards, Mike. SDN/NFV Solution Architect On 10 January 2014 16:11, Luke Gorrie wrote: > Howdy Stackers! > > We are developing a new open source Network Functions Virtualization > driver for Neutron. I am writing to you now to ask for early advice > that could help us to smoothly bring this work upstream into OpenStack > Juno. > > The background is that we are open source developers working to > satisfy the NFV requirements of large service provider networks > including Deutsche Telekom's TeraStream project [1] [2]. We are > developing a complete NFV stack for this purpose: from the DPDK-like > traffic plane all the way up to the Neutron ML2 driver. > > We are developing against Havana, we attended the Icehouse summit and > had a lot of great discussions in Hong Kong, and our ambition is to > start bringing running code upstream into Juno. > > Our work is 100% open source and we want to work in the open with the > wider OpenStack community. Currently we are in "heads-down hacking > mode" on the core functionality, but it would be wonderful to connect > with the upstream communities who we hope to be working with more in > the future (that's you guys). > > More details on Github: > https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv > > Thanks for reading! > > Cheers, > -Luke > > [1] Ivan Pepelnjak on TeraStream: > http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html > [2] Peter Löthberg's presentation on TeraStream at RIPE 67: > https://ripe67.ripe.net/archives/video/3/ > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Howdy Stackers! We are developing a new open source Network Functions Virtualization driver for Neutron. I am writing to you now to ask for early advice that could help us to smoothly bring this work upstream into OpenStack Juno. The background is that we are open source developers working to satisfy the NFV requirements of large service provider networks including Deutsche Telekom's TeraStream project [1] [2]. We are developing a complete NFV stack for this purpose: from the DPDK-like traffic plane all the way up to the Neutron ML2 driver. We are developing against Havana, we attended the Icehouse summit and had a lot of great discussions in Hong Kong, and our ambition is to start bringing running code upstream into Juno. Our work is 100% open source and we want to work in the open with the wider OpenStack community. Currently we are in "heads-down hacking mode" on the core functionality, but it would be wonderful to connect with the upstream communities who we hope to be working with more in the future (that's you guys). More details on Github: https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv Thanks for reading! Cheers, -Luke [1] Ivan Pepelnjak on TeraStream: http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html [2] Peter Löthberg's presentation on TeraStream at RIPE 67: https://ripe67.ripe.net/archives/video/3/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
