On Fri, Dec 18, 2015 at 5:26 PM, Ryan Stone <[email protected]> wrote:
> My employer is going through the process of extending our product to > support multi-tenant networking. The details of what are product does > isn't really relevant to the discussion -- it's enough to know that we have > a number of daemons acting as servers for various network protocols. > Multi-tenacy, as we've defined the feature, imposes the requirement that > our network services be able to communicate with clients from completely > independent networks. This has imposed the following new requirements on > us: > > - different tenant networks may have different DNS servers > - they may use different AAA mechanisms (e.g. one tenant uses LDAP, another > uses Kerberos, and a third also uses LDAP but a different server) > - they may use independent routing tables > - different tenant networks may use overlapping IP ranges (so we might see > two different clients from two different tenant networks with the IP > 192.168.0.1, for instance) > - traffic from different tenant networks is not guaranteed to be segregated > in any way -- it might all come in the same network interface, without any > vlan tagging or any other encapsulation that might differentiate tenant > networks > - we need to scale to thousands of tenant networks > - we will impose the requirement that our system can't be assigned the same > IP address for different tenant networks > > Our intention is to use the destination IP address of incoming packets to > determine which tenant network the packet came from (hence the requirement > of not allowing the same IP to be configured for different tenant > networks).. > > The obvious tool for meeting these requirements in FreeBSD is VIMAGE. > However, we have prototyped that approach already, and I have been told > that we discovered that we found this will not scale to thousands of > networks. I don't have all of the details as to why, but the root of the > problem is that any given process can only be associated with a single vnet > instance. In our current architecture, we can't have thousands of > instances of each network service running (and I'm not sure that we really > could: if we support A services, B tenant networks and C CPU cores, we > would need a minimum of A * B * C threads to ensure that any given service > on any single tenant network could fully utilize the system's resources to > process requests). > > > We're instead looking at using FIBs to implement the routing table > requirement. To meet our requirements, we're expecting to have to make > three important chances to how FIBs are managed: > > 1) Allow listening sockets to be wildcarded across FIBs > 2) Make FIBs a property of an interface address, not the interface > 3) Allow each thread to set a default FIB that will be applied to newly > created sockets > > 1) > We don't really want to change all of our services to instantiate one > listening socket for every tenant network. Instead we're looking at > implementing (and upstreaming) a kernel extension that allows a listening > socket to be wildcarded across all FIBs (note: yesterday I described this > feature as allowing us to pick-and-choose FIBs, but people internally have > convinced me that a wildcard match would make their lives significantly > easier). When a new connection attempt to a listening socket in this mode > is accepted, the socket would not inherit its FIB from the listening > socket. Instead, it would be set based on the local IP address of the > connection. > > 2) > Currently, FIBs are a property of an interface (struct ifnet). We aren't > very enthusiastic about the prospect of having to create thousands of > interfaces to support thousands of network interfaces. We would instead > like to make the FIB a property of the interface address. For backwards > compatibility reasons I would still let admins set a FIB on an ifnet, but > instead that would be the default FIB assigned to addresses that aren't > explicitly assigned a FIB. That should maintain the current behaviour > while making it easy to push FIBs down into the address. > > 3) > The idea of a per-thread FIB has gotten the most pushback so far, and I > understand the objection. I'll explain the problem that we're trying to > solve with this. When a new request comes in, we may need to perform > authentication through LDAP or Kerberos. The problem is that the existing > open-source implementations that we are using manage sockets directly. We > really don't want to have to go through them and make their APIs entirely > FIB-aware -- that is far too much churn. By moving awareness of the > current FIB into the kernel, existing calls to socket() can do the right > thing transparently. > > We're not entirely happy with the solution, but the "right" way to solve > the problem involves rototilling a number of libraries. Even if we could > convince the upstream projects to take patches, it's far more work than > we're willing to take on. > > Why not address (3) by creating your own wrappers of the socket library calls whose behavior you want to change (see --wrap in ld(1))? Your wrapper(s) could then implement whatever application-specific FIB selection policy you'd like, for example, checking for a particular thread local storage key whose value indicates the FIB to use, and if defined setting the FIB ahead of invoking the original socket library call. _______________________________________________ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-transport To unsubscribe, send any mail to "[email protected]"
