On 2/13/15 11:09 , Dirk Steinberg via smartos-discuss wrote: > Hi, > > I read that Joyent is working on extending SmartOS/SDC with network > virtualization based on VXLAN. I assume that in addition to a kernel > module for VXLAN interfaces there will be some control plane / > orchestration layer that works together with SDC. > > Is there some architecture documentation available that would explain > the big picture? How is the overlay network supposed to work? > Just plain basic MAC learning, or something more advanced with > an actual control plane (BGP?) like in Juniper Contrail or Alcatel > Nuage? > > Will use cases like service chaining be supported? > > Is a current (2015, with lxbrand) binary build available for testing? > The latest bits I could find are almost 5 month old. > http://dtrace.org/blogs/rm/2014/09/23/illumos-overlay-networks-development-preview-02/ > > <http://dtrace.org/blogs/rm/2014/09/23/illumos-overlay-networks-development-preview-02/> > > The userland lookup as explained in that blog post sounds promising, > I am hoping for a BGP based control plane in userland in the future… > Does anyone care to share plans and status updates?
Hi Dirk, I'm probably the best person to answer this as I'm in the coal mine. I haven't put out another preview of the bits, but I could probably get something out in the next week or two. We're currently working at getting the whole system up and into SDC. Let me start with explaining how that works, and then we'll go into how the broader SDN is designed so you can build your own and interface with other existing SDN solutions. As an aside, I had a presentation at illumos day where I talked about this: https://www.youtube.com/watch?v=mLvi3nPNqiQ&list=PLH8r-Scm3-2VmZhZ76tFPAhPOG0pvmjdA&index=3 and slides http://fingolfin.org/illumos/talks/sdn.pdf. In SDC, we're adding the notion of what we call a 'fabric'. The idea behind this fabric is that it represents the equivalent of a physical networking fabric, eg. a loosely connected series of hardware that may or may not have overlapping L2/L3 domains that's similar to the SDC sense of this. From an SDC perspective, this allows users to go and create any number of VLANs and L3 Networks on top of it, as long as it uses RFC1918 addresses. With the foundation in place, there are lots of ways one can then extend this to things like elsatic ips, etc. As part of SDC, all of the network state is stored in moray -- our key/value store backed by manatee (HA Postgres). That data is proxied and abstracted by a service we call portolan (https://github.com/joyent/sdc-portolan). This service is still in development, but the idea is that we have a small TCP based protocol that the individual compute nodes speak to talk to these portolan instances which in turn talk to the key-value store. Their purpose is to abstract the backing store and allow us a bit more flexibility in terms of upgrade. You can read more about that API here: https://github.com/joyent/sdc-portolan/blob/master/docs/varp. Of course, this brings us to the individual architecture in illumos. The core philosophy of the illumos architecture is that logic in encap/decap should be separate from how we do look ups. The README here: https://github.com/joyent/illumos-joyent/tree/dev-overlay explains a bit of what's going on. They key here is that when the kernel tries to send to a link layer address it doesn't know (generally a MAC), it goes and does a call to the varpd daemon in userland. It does this in an asynchronous fashion that generally resembles ARP, hence why we called it varpd - the virtual arp daemon. >From there, there's a plug-in interface that one can implement to define how you do look ups. The documentation is a bit sparse, but the API is here: https://github.com/joyent/illumos-joyent/blob/dev-overlay/usr/src/lib/varpd/libvarpd/common/libvarpd_provider.h. Currently, I've implemented three different plugins for this. The first is a simple point to point plugin, eg. it's basically just a glorified tunnel, send everything over here -- https://github.com/joyent/illumos-joyent/tree/dev-overlay/usr/src/lib/varpd/direct. The second is basically a glorified form of /etc/ethers, basically a file plugin that you can use as a static form of defining dynamic lookups -- https://github.com/joyent/illumos-joyent/tree/dev-overlay/usr/src/lib/varpd/files. Finally, the last and most complicated example is what we've done for SDC, which you can see here: https://github.com/joyent/illumos-joyent/tree/dev-overlay/usr/src/lib/varpd/svp. So, given all this, it allows you to go ahead and figure out how you want to interact with it and go from there. If someone wants something that goes and interacts with some existing system or does it in some other way, it should hopefully be able to fit into this model and you'd have to define and write your own plugin. There's a lot here, so let me know what questions you have and what I can better clarify. Thanks, Robert ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
