On Tue, Jan 14, 2014 at 05:22:03PM +0200, Vangelis Koukis wrote: > On Mon, Jan 13, 2014 at 02:39:32pm +0100, Jose A. Lopes wrote: > > On Fri, Jan 10, 2014 at 05:50:32PM +0200, Vangelis Koukis wrote: > > [snip] > > > In a similar note, who will be responsible for setting up the DHCP > > > server? It could be the administrator's responsibility, but then if it > > > is Ganeti the entity which picks the MAC addresses and IPs for the guest > > > side of the TAP interfaces, how will this DHCP server be notified, so as > > > to only server the correct IP addresses to specific MAC addresses? > > > > > > Also, if it is the administrator's responsibility, then perhaps the > > > admin should be able to set up standard ifup hooks, like for every > > > other interface of an instance. But in the following examples, you > > > specifically set script=no,downscript=no. > > > > > > Another possibility would be for Ganeti to come prepackaged with its > > > own, embedded DHCP server just for serving requests on the TAPs used for > > > the communication mechanism. We've been using snf-nfdhcpd > > > (https://code.grnet.gr/projects/snf-nfdhcpd) for just that in > > > production. > > > > > > Actually, in previous conversation Guido had asked us to document how to > > > set it up with Ganeti, and merge the resulting docs with the Ganeti > > > upstream. Perhaps it would make sense to combine the effort now, and use > > > snf-nfdhcpd as an embedded DHCP server with Ganeti. Sorry for not having > > > documented it earlier. > > > > And by the way, just out of curiosity, what were the reasons that lead > > you to implement your own DHCP server? Have considered other servers? > > Have you tried, for example, dnsmasq? > > > > Hello Jose, > > I'll try to summarize here the main principles behind snf-nfdhcpd, > and its differences compared to dnsmasq or similar servers. > > snf-nfdhcpd is based on NFQUEUE. The administrator can enable processing > of DHCP requests on individual TAP interfaces by injecting snf-nfdhcpd > in the processing pipeline for IP packets dynamically. > > The daemon runs on the host and is controlled by manipulating files under its > state directory, /var/lib/nfdhcpd by default (configurable). Creation of a new > file under this directory ("binding file") instructs the daemon to answer DHCP > requests on the specified TAP interface. Please see below for an example > binding file. > > A simple scenario is: > a) snf-nfdhcpd starts. Upon initialization, it creates an NFQUEUE (e.g., 42, > configurable), and listens on it for incoming DHCP requests. It also begins to > watch its state directory, /var/lib/nfdhcpd via inotify(). > b) A new VM gets created, let's assume its NIC has address mac0, lives on TAP > interface tap0, and is to receive IP address ip0 via DHCP. > c) Someone (e.g., a Ganeti KVM ifup script, or in our case snf-network, > see http://code.grnet.gr/projects/snf-network) creates a new binding > file informing snf-nfdhcpd that it is to reply to DHCP requests from MAC > mac0 on TAP interface tap0, and include IP ip0 in the DHCP reply. > d) The administrator injects snf-nfdhcpd in the processing pipeline for > packets > coming from tap0, using iptables. This can happen for every TAP > interface, e.g.: > # iptables -t mangle -A PREROUTING -i tap+ -m udp -p udp --dport 67 -j > NFQUEUE --queue-num 42
We deciced to use the DHCP server to avoid iptables, given that existing installations probably already have complicated iptables setups. This way, we can avoid adding to the confusion. By the way, avoiding iptables was actually a request from Apollon. If we were to use iptables, we could probably get by by throwing away the DHCP server and using NAT instead to replace any source IP address of incoming packets from the TAP interfaces with a unique IP address within the host. Is there a way to configure snf-nfdhcpd without iptables? > or for individual TAP interfaces. > e) From now on, whenever a DHCP request is sent out by the VM, the > iptables rule will forward the packet to nfdhcpd, which will consult > its bindings database, find the entry for tap0, verify the source MAC, > and inject a DHCP reply for the corresponding IP address into tap0. > > This has various advantages compared to dnsmasq or similar servers: > > a) The DHCP service can be activated dynamically, per-interface, > by manipulating iptables accordingly. There is no need to restart > the daemon, or edit (potentially read-only) configuration files, > you only need to drop a file under /var/lib/nfdhcpd. > > b) There is no interference to existing DHCP servers listening to port > 67. Everything happens directly via NFQUEUE. > > c) The host doesn't even need to have an IP address on the interfaces > where DHCP replies are served, making it invisible to the VMs. This > may be beneficial from a security perspective. Similarly, it doesn't > matter if the TAP interface is bridged or routed. > > d) MAC addresses are bound on TAP interfaces. Requests coming from > unrelated TAP interfaces are ignored, and packet processing happens > as if snf-nfdhcpd didn't exist in the first place. > > e) snf-nfdhcpd is written in pure Python and uses scapy for packet > processing. This has proved super-useful when trying to troubleshooting > networking problems in production. > > Example snf-nfdhcpd binding file: > A binding file in snf-nfdhcpd's state directory is named after the > physical interface where the daemon is to receive incoming DHCP requests > from, and defines at least the following variables: > > * INDEV: The logical interface where the packet is received on. For > bridged setups, the bridge interface, e.g., br0. Otherwise, same as > the file name. > * MAC: The MAC address where the DHCP request must be originating from > * IP: The IPv4 address to be returned in DHCP replies > * SUBNET: The IPv4 subnet to be returned in DHCP replies in CIDR form > * GATEWAY: The IPv4 gateway to be returned in DHCP replies > > Please see package snf-network for an example Ganeti KVM ifup script > which exercises all of the above, plus IPv6 autoconfiguration: > > https://code.grnet.gr/projects/snf-network/repository/revisions/master/entry/common.sh#L153 > > We are in the process of integrating the above description as > documentation for snf-nfdhcpd. > > Looking forward to your comments, > > Thanks, > Vangelis. -- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
