On 28/11/12 16:48, Gary Kotton wrote: > On 11/28/2012 04:33 PM, Mike Kolesnik wrote: >> ----- Original Message ----- >>> On 11/28/2012 03:46 PM, Livnat Peer wrote: >>>> On 28/11/12 14:00, Gary Kotton wrote: >>>>> On 11/28/2012 01:34 PM, Livnat Peer wrote: >>>>>> On 27/11/12 16:34, Gary Kotton wrote: >>>>>>> On 11/27/2012 04:06 PM, Mike Kolesnik wrote: >>>>>>>> Thanks for the reply, >>>>>>>> Please see comments inline >>>>>>>> >>>>>> Hi Garry, >>>>>> Thanks for your input, see my comments inline. >>>>>> >>>>>> Livnat >>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> On 11/27/2012 03:01 PM, Livnat Peer wrote: >>>>>>>>>> Hi All, >>>>>>>>>> Mike Kolesnik and me have been working on a proposal for >>>>>>>>>> integrating >>>>>>>>>> quantum into oVirt in the past few weeks. >>>>>>>>>> We decided to focus our efforts on integrating with quantum >>>>>>>>>> services, we >>>>>>>>>> started with IP address management service. >>>>>>>>>> >>>>>>>>>> here is a link to our proposal: >>>>>>>>>> http://wiki.ovirt.org/wiki/Quantum_IPAM_Integration >>>>>>>>>> >>>>>>>>>> As usual comments are welcome, >>>>>>>>> Please see my comments below: >>>>>>>>> >>>>>>>>> i. The quantum diagram is incorrect. It is the same message >>>>>>>>> queue >>>>>>>>> that >>>>>>>>> passes the notifications. This is done by a message broker. In >>>>>>>>> RH we >>>>>>>>> are >>>>>>>>> supporting qpid and in the community upstream rabbitmq is >>>>>>>>> used. >>>>>>>> I will fix the diagram accordingly >>>>>>> Thanks >>>>>>>>> ii. The DHCP agent is plugable. That is there may be more than >>>>>>>>> one >>>>>>>>> implementation. At the moment only dnsmasq is support. There >>>>>>>>> was a >>>>>>>>> company working on ISC upstream but they have stopped due to >>>>>>>>> problem >>>>>>>>> they encountered. >>>>>>>>> iii. Layer 3 driver. This is incorrect. The layer 2 agent does >>>>>>>>> the >>>>>>>>> network connectivity. The layer 3 agent provides floating IP >>>>>>>>> support. >>>>>>>>> This is something that you may want to consider to. It is >>>>>>>>> related to >>>>>>>>> IPAM >>>>>> From what we gathered from code the DHCP Agent is >>>>>> communicating with (an >>>>>> implementation of the) LinuxInterfaceDriver, which is not the >>>>>> same as >>>>>> the layer 2 agent used in the plugin. >>>>> Correct. The DHCP agent needs to create the relevant interfaces. >>>>> The >>>>> layer 2 is responsible for attaching these interfaces to the >>>>> network. >>>>> >>>>>> For example, looking in Linux bridge, the plugin has >>>>>> Linux_bridge_quantum agent that is part of the Linux bridge >>>>>> plugin, and >>>>>> it has (what we called Layer 3 driver) a BridgeInterfaceDriver >>>>>> that is >>>>>> used within the DHCP Agent. >>>>>> >>>>>> Maybe we used a misleading terminology but 'layer 2 agent' is >>>>>> also >>>>>> misleading, IMO, as it is already used in the plugin context and >>>>>> this is >>>>>> not the same component. >>>>>> >>>>>> We'll update the doc to call it 'layer 2 driver'. >>>>>> >>>>>>>>> iv. I am not really sure I understand you picture with server >>>>>>>>> B and >>>>>>>>> get/create network. This is not really what happens. If you >>>>>>>>> want I >>>>>>>>> can >>>>>>>>> explain. >>>>>>>> We saw that the DHCP Agent is trying to create the network >>>>>>>> interface >>>>>>>> if it doesn't exist (in DeviceManager.setup which is called as >>>>>>>> part of >>>>>>>> "enable_dhcp_helper"). >>>>>>>> >>>>>>>> If you want to elaborate on this, please do. >>>>>>> The DHCP agent will create a device that is used by the dnsmasq >>>>>>> process. >>>>>>> The creation is done according to a driver that is used for the >>>>>>> underlying l2 implementation. It does not have anything to do >>>>>>> the the >>>>>>> layer 3 agent. >>>>>> Again the same terminology misunderstanding. >>>>>> >>>>>>> It creates a network device and assigns it an IP address. >>>>>>> The layer 2 agent (if there is one) will attach this device to >>>>>>> the >>>>>>> underlying virtual network. >>>>>> It seems to be our understanding and what we have described in >>>>>> the wiki >>>>>> page, do you see something wrong there? >>>>>> >>>>>>> Prior to doing anything the DHCP agent will create a quantum >>>>>>> port on the >>>>>>> subnet. This is how it receives its own IP address. >>>>>>> >>>>>>>>> v. What do you mean by the "port is then part of the Quantum >>>>>>>>> DB". Not >>>>>>>>> all plugins maintain a database. >>>>>>>> True but if it's not saved somewhere then how does the Agent >>>>>>>> know >>>>>>>> which IP to assign to which MAC? >>>>>>> The DHCP agent is notified by the Quantum service of a new port >>>>>>> allocation. It is passed the port details - the mac address and >>>>>>> the IP >>>>>>> address. The plugin may not use a database that one can access. >>>>>>> All of >>>>>>> the interface to the data is done via the Quantum API. For >>>>>>> example >>>>>>> the NVP. >>>>>>> >>>>>>>>> vi. I think that you are missing useful information about the >>>>>>>>> subnets >>>>>>>>> and gateways. This is also a critical part of the IPAM. When a >>>>>>>>> VM >>>>>>>>> sends >>>>>>>>> a DHCP request it not only gets and IP but it can also receive >>>>>>>>> host >>>>>>>>> route information. This is very important. >>>>>>>> can you please elaborate on this? >>>>>>> When you reboot your computer at work you get access to the >>>>>>> internet. >>>>>>> This is done via DHCP. You get an IP address and all of the >>>>>>> relevant >>>>>>> routes configured. The port data has the 'host_routes' which is >>>>>>> also >>>>>>> used by the dnsmasq. There can be more than one route which is >>>>>>> configured. The subnet contains the gateway IP. >>>>>>> >>>>>> We assumed that when creating the subnet in Quantum it would >>>>>> update the >>>>>> DHCP Agent with all the information oVirt will provide as part of >>>>>> the >>>>>> subnet details (dns_nameservers, host_routes, gateway_ip etc). >>>>>> Isn't this the case? >>>>> Yes it is. I was misleading as the wiki only referred to Quantum >>>>> ports >>>>> and not subnets. If I understand correctly then you will be using >>>>> the >>>>> entire Quantum service? Will this include floating IP's security >>>>> groups >>>>> etc.? >>>> I did not review security group and floating IP provided by >>>> quantum. Can >>>> you please point us to a documentation on these? >>> Security groups is in development - >>> https://blueprints.launchpad.net/quantum/+spec/quantum-security-groups >>> Floating IP's - >>> http://docs.openstack.org/trunk/openstack-network/admin/content/index.html >>> >>>> >>>>>>>>> vii. The DHCP agent dynamics are incorrect (l3 agent, write >>>>>>>>> port >>>>>>>>> definitions etc.). One of the pain points is that the process >>>>>>>>> is for >>>>>>>>> each quantum network. This is a scale issue and is being >>>>>>>>> discussed >>>>>>>>> upstream. >>>>>>>> This is what we saw that happens in the code, if we are wrong >>>>>>>> please >>>>>>>> explain what is the right behaviour of the DHCP Agent. >>>>>>> For each network that has one or more subnets with DHCP support >>>>>>> a >>>>>>> dnsmasq process is created. Please see http://fpaste.org/IHbA/. >>>>>>> Here I >>>>>>> have two networks. >>>>>> That's exactly what we have described in the wiki. dnsmasq per >>>>>> network. >>>>>> In the integration with oVirt we planned that the ovirt layer2 >>>>>> driver >>>>>> will not return interface_name where there is no need for the >>>>>> dnsmasq >>>>>> locally on the host. >>>>> I do not think that this will work - you will need to attach the >>>>> dnsmasq >>>>> to the network. At the moment Quantum does not run the dnsmasq on >>>>> the >>>>> compute nodes. There is a notion of a network node when various >>>>> services >>>>> can run. One of the issues that we are dealing with at the moment >>>>> is HA >>>>> and scale for the DHCP agents. At the moment only one DHCP agent >>>>> can >>>>> run. The open issue is that if the DHCP agent sees thousands of >>>>> networks >>>>> then it will create dnsmasq process for each network killing the >>>>> node >>>>> local resources. >>>>> >>>> That's exactly the problems we addressed in our proposal. >>>> In the integration proposal we'll deploy the DHCP-Agent on the >>>> hosts >>>> (where and when is defined via the the setupDHCP API we added to >>>> oVirt), >>>> and we'll have more than one DHCP Agent...each DHCP Agent will >>>> manage >>>> the networks available on the host it is deployed on. >>> At the moment Quantum only works with 1 DCHP agent. Upstream there is >>> work on improving this. >> So if I run Quantum Service and 2 instances of the DHCP Agent on >> different machines, this is not supported? (And it it's not can you >> please elaborate what is the problem?) > > There is a bug in the underlying Oslo RPC implementation that sets the > topic and queue names to be same value. Hence it will only be sent to > one agent and not all.
yes, and it seems it to also bothers Mark McClain which wrote on the openstack-dev list - " There is a bug in the underlying Oslo RPC implementation that sets the topic and queue names to be same value. I didn't get a clear explanation of this problem until today and will have to figure out a fix to oslo." either he'll fix it or we can add that to our list of required fixes to Quantum. > > >> >>>> The wiki page describes how we intend to do it. >>>> We actually leverage the fact that the DHCP Agent and the Quantum >>>> service don't have to be co-located on the same machine and added >>>> some >>>> logic in oVirt where and when to deploy the DHCP Agent. >>>> >>>> The DHCP Agent will get notification on each network created in >>>> Quantum >>>> but when delegating the call to the layer 2 (oVirt) driver we'll >>>> create >>>> devices only for the networks we'd like to control from that DHCP >>>> instance. >>> Not sure what the layer 2 oVirt driver is? Does this mean that you >>> will >>> not be using Quantum L2 agents? If so then this may not work. First I >>> need a clarification then I can explain. >>> >>>> In case we won't create a device we would like the DHCP Agent >>>> to avoid spawning a dnsmasq (which is the code we'll contribute to >>>> Quantum). >>> The DHCP agent creates the device. I do not understand how you will >>> decide whether or not to create the device. One thing that you should >>> take into account is HA for the solution. Lets say your DHCP agent on >>> the node freezes - how will launched VM's get their IP addresses? >>>>>> This requires a patch to Quantum that in case the driver returns >>>>>> empty >>>>>> device name the dnsmasq won't be started. >>>>> I am not sure that I understand. The DHCP agent has to create a >>>>> device >>>>> to interface with the outside world. If the device fails to be >>>>> created >>>>> then the dnsmasq process will not be spawned. >>>>> >>>> If the device fails to to be created and an exception is raised >>>> then the >>>> dnsmasq is not created (there is a retry in a loop via the error >>>> handling in the notification layer), but we suggest that if the >>>> device >>>> returns an empty device name the DHCP Agent won't spawn a dnsmasq >>>> process as it would have no meaning. >>> ok. I am not sure how this will be accepted upstream as the DHCP >>> agent >>> is requesting to create the interface. The logic seems a tad odd. >>> As mentioned above there is work upstream on this at the moment - >>> there >>> are a number of options that are in debate - one is to have scheduler >>> that decides where to run the agents. Another is to indicate the the >>> DHCP agent which networks to treat - that is, if it receives a >>> network >>> notifiction for a network that it does not "own" then it will ignore >>> this. >> Yes this is essentially our proposal.. > > ok, this was certainly not clear from the wiki :) > >> >>>> We'll use the above behavior in oVirt driver to control which >>>> dnsmask is >>>> spawned on the server the DHCP Agent is deployed on. >>>> >>>>>> We'll send a patch for that soon. >>>>>> I added that to the wiki as well. >>>>>> >>>>>>>>> viii. Quantum does not require homogeneous hardware. This is >>>>>>>>> incorrect. >>>>>>>>> There is something called a provider network that addresses >>>>>>>>> this. >>>>>>>> Can you please elaborate? >>>>>>> When you create a network you can indicate which NIC connects to >>>>>>> the >>>>>>> outside world. If you look at >>>>>>> http://wiki.openstack.org/ConfigureOpenvswitch then you will see >>>>>>> the >>>>>>> bridge mappings. This information is passed via the API. >>>>>> Our understanding is that Quantum IPAM design assumes the DHCP >>>>>> Agent has >>>>>> local access to *ALL* the networks created in quantum. >>>>> IPAM is part of the Quantum API. That is, Quantum provides and >>>>> interface >>>>> for logical ports to be assigned an IP address. The DHCP agent is >>>>> one >>>>> way of implementing this. The DHCP agent interfaces with the >>>>> Quantum >>>>> plugin to receive the information that it requires. Currently tyhe >>>>> DHCP >>>>> agent is able to get information for all networks. >>>>> >>>>>> Per Network it spawns a local dnsmasq and connect it to the >>>>>> network >>>>>> (which should be accessible from within the host on which the >>>>>> DHCP Agent >>>>>> is running on). >>>>> The dnsmasq is able to be accessed from all compute nodes on the >>>>> network. From what you are mentioning here is that you guys will >>>>> be >>>>> taking a hybrid approach to using Quantum. Correct? >>>>> >>>> I did not understand the question, not sure what you mean by hybrid >>>> approach? >>> From what you have written I understand, and may be completely wrong >>> here, that you only want to use certain parts of quantum in certain >>> ways >>> which are not supported today. So the hybrid is taking parts of >>> Quantum >>> and using them in VDSM but not via the standard API's. >> We are planning to write our own pluggable implementations (which is >> the purpose of Quantum), but we're not planning to take just parts of >> Quantum but the whole deal.. > > Can you please elaborate more on the plugin that you plan to implement. > Will this be contributed upstream? If so then I suggest that you guys > draft blueprints The plugin would obviously be open source and we are working on the details ATM. I don't see a reason we won't contribute the code to quantum - The code is open anyway. >> >>>>>> This assumption is problematic in the oVirt context and this is >>>>>> the >>>>>> issue we were trying to overcome in the proposed integration. >>>>> I am sorry but I am not sure that I understand the issue that you >>>>> are >>>>> trying to overcome. >>>> It's the same issues you raised above, scalability and the >>>> assumption >>>> that there is one hast that has to have connectivity to all the >>>> networks >>>> configured in the system. >>>> >>>>> In theory more than one DHCP server can run. This is >>>>> how people provide HA. One of the servers will answer. Do you plan >>>>> to >>>>> have a DHCP agent running on each vdsm node? Nova networking has >>>>> support >>>>> for a feature like this. It is called multinode. It is something >>>>> that is >>>>> under discussion in Quantum. >>>> The issue is not related to the DHCP Agent HA. >>> I am not sure how your solution will address the HA. Say you have 2 >>> hosts. VM X is running on HOST A. It has a dnsmasq running on HOST A. >>> HOST B will not have one as there are no VM's running on B. Say the >>> dnsmasq freezes on A. A new VM deployed on A will not receive and IP >>> address. If there was a dnsmasq running on B Then it would. >> We are planning on having several DHCP servers for each network, so >> this will not be a problem. > > Great. >> >>>>>>>>> ix. I do not udnerstand the race when the VM starts. There is >>>>>>>>> none. >>>>>>>>> When >>>>>>>>> a VM starts it will send a DHCP request. If it does not >>>>>>>>> receive one >>>>>>>>> it >>>>>>>>> will send another after a timeout. Can you please explain the >>>>>>>>> race? >>>>>>>> This is exactly it, the VM might start requesting DHCP lease >>>>>>>> before it >>>>>>>> was updated in the DHCP server, for us it's a race. >>>>>>> This works. This is how DHCP is engineered. Can you please >>>>>>> explain the >>>>>>> problem? If you send a DHCP request and do not get a reply then >>>>>>> you send >>>>>>> one again. The timeout between requests is incremental. >>>>>>> >>>>>>> I am not sure that we are on the same page when it comes to a >>>>>>> race >>>>>>> condition. I'd like you to clarify. >>>>>>>>> You do not need to consume Quantum to provide IPAM. You can >>>>>>>>> just run >>>>>>>>> the >>>>>>>>> dnsmasq and make sure that its interface is connected to the >>>>>>>>> virtual >>>>>>>>> network. This will provide you with the functionality that you >>>>>>>>> are >>>>>>>>> looking for. If you want I go can over the dirty details. It >>>>>>>>> will be >>>>>>>>> far >>>>>>>>> less time than consuming Quantum and you can achieve the same >>>>>>>>> goal. >>>>>>>>> You >>>>>>>>> just need to be aware when the dnsmasq is running to sent the >>>>>>>>> updates. >>>>>>>>> >>>>>>>>> IPAM is one of the many features that Quantum has to offer. It >>>>>>>>> will >>>>>>>>> certain >>>>>>>>> help oVirt. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Gary > _______________________________________________ Arch mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/arch
