Re: First shot at adding IPMI to qemu
I haven't heard anything about these patches. Any comments, good or bad? Has anyone tried these? Thanks, -corey On 07/02/2012 02:44 PM, miny...@acm.org wrote: I had asked about getting an IPMI device into qemu and received some interest, and it's useful to me, so I've done some work to add it. The following patch set has a set of patches to add an IPMI KCS device, and IPMI BT device, a built-in BMC (IPMI management controller), and a way to attach an external BMC through a chardev. There was some discussion on whether to make the BMC internal or external, but I went ahead and added both. The internal one is fairly basic and not extensible, at least without adding code. I've modified the OpenIPMI library simulator to work with the external interface to allow it to receive connections from the qemu external simulator with a fairly basic protocol. I've also added the ability for the OpenIPMI library to manage a VM to power it on, power it off, reset it, and handle an IPMI watchdog timer. So it looks quite like a real system. Instructions for using it are in the OpenIPMI release candidate I uploaded to https://sourceforge.net/projects/openipmi Since IPMI can advertise it's presence via SMBIOS, I added a way for a driver to add an SMBIOS entry. I also added a way to query a free interrupt from the ISA bus, since the interrupt is in the SMBIOS entry and nobody really cares which one is used. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Adding an IPMI BMC device to KVM
On 05/18/2012 08:08 AM, Stefan Hajnoczi wrote: On Mon, May 7, 2012 at 3:30 PM, Anthony Liguori wrote: On 05/06/2012 09:39 AM, Avi Kivity wrote: On 05/06/2012 05:35 PM, Anthony Liguori wrote: So what's really the use case here? Would an IPMI -> libvirt bridge get you what you need? I really think that's the best path forward. Sorry for sending this twice. I thought my mailer was set to disable HTML. Many of our customer use the standard watchdog timer and power controls on IPMI. Basically, the watchdog timer needs to be able to power off, power cycle, reset, and send an NMI. The IPMI watchdog can be configured to do all those things. (The NMI is for a "pretimeout" that generally triggers a panic.) Power control is no big surprise. It's true that you have ACPI to do this, but that's not terribly useful on non-x86 (and non ia64, I suppose) systems. Do you want to expose host sensors - the challenge is that they don't reflect the hardware that the virtual machine sees? Or do you want to have synthetic sensors - which virtual sensors are useful to have? A few sensors, notably the watchdog timer sensor, are not synthetic. The others generally are. Sensors cover things beyond just power and temperature. Important other ones in IPMI deal with the presence of FRUs in the system, BIOS/OS state, and intrusion detection. Two main reasons for synthesized sensors exist. One, in a legacy situation, is to "fool" the management system into thinking everything is ok, since it is expecting to see these sensors with specific values. You could even possibly reflect the state of real sensors, mapping them somehow, if you wanted. The other reason is for testing. It's pretty hard to induce some of these sensors to go out of range in a real system. So without simulation, you need a hack-ed up management controller in a real system to truly test your software. Is the IPMI watchdog useful, QEMU already supports the i6300esb PCI watchdog (see qemu -watchdog option documentation)? That's only useful if you can simulate an i6300esb. Probably not possible on non-x86. Plus legacy systems may be expecting the capabilities of the IPMI watchdog. Some use cases that illustrate how the guest is going to use IPMI would be interesting and could help guide the discussion. You are probably right. I've mentioned a couple above. One other possible one, that someone else mentioned, is the ability to control a VM using standard tools like ipmitool or OpenIPMI over network interfaces. People may already have management systems that are designed around IPMI, and it would make a move to virtual machines easier. This was the request mentioned that was in the Redhat database. Such a capability would require re-thinking things a bit. You have two basic options that I see. Either have the "management controller" run outside qemu and define a simple interface to it, or modify qemu to be able to run the management controller internally. My preference in to run it outside qemu, for the following reasons: * It decouples things that are IPMI internals from having to go through getting into qemu repositories. * The management controller then becomes useful for a number of other purposes. I already have one mostly done that I use for testing. It could be used by other VMs. * A full management controller is a fairly big piece of software, especially if you include the network access. It probably doesn't belong in qemu. * No offense, I don't want to muck around inside qemu to accomplish this :). There are, of course, disadvantages. Some I can think of: * There can be confusion about which versions of two things work together. * Overall, it is more complex to get working. * There are possible security implications. So I guess those are points we can talk about... -corey -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fwd: Re: Adding an IPMI BMC device to KVM
Resending to the list, plan text only. Sorry about that... Original Message Subject:Re: Adding an IPMI BMC device to KVM Date: Mon, 07 May 2012 16:57:06 -0500 From: Corey Minyard Reply-To: miny...@acm.org To: Anthony Liguori CC: kvm@vger.kernel.org, Avi Kivity On 05/05/2012 09:29 AM, Anthony Liguori wrote: You could just use a USB rndis device and then run an IPMI server on the host. That is probably the simpliest way to test. I'm trying to figure out how RNDIS would be helpful here. You can't have the guest talk USB for this, it's going to talk the standard interfaces. However, I am loathe to create my own protocol for talking between the BMC and QEMU. Perhaps RNDIS could be useful, but it's massive overkill. It also doesn't provide any security. If there were client and server libraries that would run over a socket, that would be tempting. I'm wondering if security is that big a deal, though. If you use a chardev, and you have QEMU make a connection to the BMC, maybe that would be ok. BMCs typically run a full OS like Linux making emulation as a device in QEMU prohibitively hard. BMCs are generally very simple 8-bit microcontrollers, unless they are something like an ATCA shelf manager. Emulation is pretty easy. Well, emulation of an ATCA shelf manager wouldn't be easy, but that's hopefully not required in the near future. -corey Regards, Anthony Liguori On May 4, 2012 2:10 PM, "Corey Minyard" <mailto:tcminy...@gmail.com>> wrote: > > I've been working on adding an IPMI BMC as a virtual device under KVM. I'm > doing this for two primary reasons, one to have a better test environment than > what I have now for testing IPMI issues, and second to be able to better > simulate a legacy environment for customers porting legacy software. > > For those that don't know, IPMI is a system management interface. Generally > systems with IPMI have a small microcontroller, called a BMC, that is always on > when the board is powered. The BMC is capable of controlling power and reset > on the board, and it is hooked to sensors on the board (voltage, current, > temperature, the presence of things like DIMMS and power supplies, intrusion > detection, and a host of other things). The main processor on a system can > communicate with the BMC over a device. Often these systems also have a LAN > interface that lets you control the system remotely even when it's off. > > In addition, IPMI provides access to FRU (Field Replaceable Unit) data that > describes the components of the system that can be replaced. It also has data > records that describe the sensor, so it is possible to directly interpret the sensor > data and know what the sensor is measuring without outside data. > > I've been struggling a bit with how to implement this. There is a lot of > configuration information, and you need ways to simulate the sensors. This type > of interface is a little sensitive, since it has direct access to the reset and power > control of a system. > > I was at first considering having the BMC be an external program that KVM > connected to over a chardev, with possibly a simulated LAN interface. This has > the advantage that the BMC can run even when KVM is down. It could even > start up KVM for a "power up", though I'm not sure how valuable that would be. > Plus it could be used for other virtual machines. However, that means there is > an interface to KVM over a chardev that could do nasty things, and even be a > possible intrusion point. It also means there is a separate program to maintain. > > You could also include the BMC inside of KVM and run it as a separate thread. > That way there doesn't have to be an insecure interface. But the BMC will need > a lot of configuration data and this will add a bunch of code to KVM that's only > tangentially related to it. And you would still need a way to simulate setting > sensors and such for testing things. > > Either way, is this interesting for including into KVM? Does anyone have any > opinions on the possible ways to implement this? > > Thanks, > > -corey > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majord...@vger.kernel.org <mailto:majord...@vger.kernel.org> > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Adding an IPMI BMC device to KVM
On 05/07/2012 02:45 PM, Dave Allan wrote: FWIW, the idea of an IPMI interface to VMs was proposed for libvirt not too long ago. See: https://bugzilla.redhat.com/show_bug.cgi?id=815136 Well, it wouldn't be to hard to do. I already have working emulation code that does the IPMI LAN interface (including the IPMI 2.0 stuff for more reasonable security). I have a KCS interface and a minimal IPMI controller working in KVM, though I'm not quite sure the best final way to hook it in. Configuration is going to be the hardest part, but a minimal configuration for providing basic management would be easy. -corey Dave On Mon, May 07, 2012 at 01:07:45PM -0500, Corey Minyard wrote: I think we are getting a little out of hand here, and we are mixing up concepts :). There are lots of things IPMI *can* do (including serial access, VGA snooping, LAN access, etc.) but I don't see any value it that. The main thing here is to emulate the interface to the guest. OOB management is really more appropriately handled with libvirt. How the BMC integrates into the hardware varies a *lot* between systems, but it's really kind of irrelevant. (Well, almost irrelevant, BMCs can provide a direct I2C messaging capability, and that may matter.) A guest can have one (or more) of a number of interfaces (that are all fairly bad, unfortunately). The standard ones are called "KCS", "BT" and "SMIC" and they generally are directly on the ISA bus, but are in memory on non-x86 boxes (and on some x86 boxes) and sometimes on the PCI bus. Some systems also have interfaces over I2C, but that hasn't really caught on. Others have interfaces over serial ports, and that unfortunately has caught on in the ATCA world. And there are at least 3 different basic types of serial port interfaces with sub-variants :(. I'm not sure what the USB rndis device is, but I'll look at it. But there is no IPMI over USB. The big things a guest can do are sensor management, watchdog timer, reset, and power control. In complicated IPMI-based systems like ATCA, a guest may want to send messages through its local IPMI controller to other guest's IPMI controllers or to a central BMC that runs an entire chassis of systems. So that may need to be supported, depending on what people want to do and how hard they want to work on it. My proposal is to start small, with just a local interface, watchdog timer, sensors and power control. But have an architecture that would allow external LAN access, tying BMCs in different qemu instances together, perhaps serial over IPMI, and other things of that nature. -corey On 05/07/2012 10:21 AM, Anthony Liguori wrote: On 05/07/2012 10:11 AM, Avi Kivity wrote: On 05/07/2012 05:55 PM, Anthony Liguori wrote: For all intents and purposes, the BMC/RSA is a separate physical machine. That's true for any other card on a machine. It has a separate power source for all intents and purposes. If you think of it in QOM terms, what connects the nodes together ultimately is the "Vcc" pin that travels across all devices. The RTC is a little special because it has a battery backed CMOS/clock but it's also handled specially. And we fail to emulate it correctly as well, wrt. alarms. The BMC does not share Vcc. It's no different than a separate physical box. It just shares a couple buses. It controls the main power place, reset line, can read VGA and emulate keyboard, seems pretty well integrated. Emulating the keyboard is done through USB. How the VGA thing works is very vendor dependent. The VGA snooping can happen as part of the display path (essentially connected via a VGA cable) or it can be a side-band using a special graphics adapter. I think QEMU VNC emulation is a pretty good analogy actually. That is one way to do it. Figure out the interactions between two different parts in a machine, define an interface for them to communicate, and split them into two processes. We don't usually do that; I believe your motivation is that the two have different power domains (but then so do NICs with wake-on-LAN support). The power still comes from the PCI bus. Maybe. But it's on when the rest of the machine is off. So Vcc is not shared. That's all plumbed through the PCI bus FWIW. Think of something like a blade center. Each individual blade does not have it's own BMC. There's a single common BMC that provides an IPMI interface for all blades. Yet each blade still sees an IPMI interface via a USB rndis device. You can rip out the memory, PCI devices, etc. from a box while the Power is in and the BMC will be unaffected. At any rate, you would have some sort of virtual hardware device that essentially spoke QMP to the slave instance. You could just do virtio-serial and call it a day actually. Sorry I lost you. Which is the master and which is the slave? The BMC is the maste
Re: Adding an IPMI BMC device to KVM
On 05/07/2012 10:17 AM, Zhi Yong Wu wrote: Then should we also emulate one AMM virtual device? one fsp virtual device? one IVE virtual device? I have no idea what those things are, so I don't know. I don't think I know of any need for them. But vendors have all kinds of crazy names for IPMI. If they are IPMI management controllers, they idea is to allow you to configure the virtual IPMI interface to have the SDRs, sensors, FRU data, etc. that you need to emulate them. That configuration is non-trivial; writing SDRs really sucks. -corey -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] Adding an IPMI BMC device to KVM
I think we are getting a little out of hand here, and we are mixing up concepts :). There are lots of things IPMI *can* do (including serial access, VGA snooping, LAN access, etc.) but I don't see any value it that. The main thing here is to emulate the interface to the guest. OOB management is really more appropriately handled with libvirt. How the BMC integrates into the hardware varies a *lot* between systems, but it's really kind of irrelevant. (Well, almost irrelevant, BMCs can provide a direct I2C messaging capability, and that may matter.) A guest can have one (or more) of a number of interfaces (that are all fairly bad, unfortunately). The standard ones are called "KCS", "BT" and "SMIC" and they generally are directly on the ISA bus, but are in memory on non-x86 boxes (and on some x86 boxes) and sometimes on the PCI bus. Some systems also have interfaces over I2C, but that hasn't really caught on. Others have interfaces over serial ports, and that unfortunately has caught on in the ATCA world. And there are at least 3 different basic types of serial port interfaces with sub-variants :(. I'm not sure what the USB rndis device is, but I'll look at it. But there is no IPMI over USB. The big things a guest can do are sensor management, watchdog timer, reset, and power control. In complicated IPMI-based systems like ATCA, a guest may want to send messages through its local IPMI controller to other guest's IPMI controllers or to a central BMC that runs an entire chassis of systems. So that may need to be supported, depending on what people want to do and how hard they want to work on it. My proposal is to start small, with just a local interface, watchdog timer, sensors and power control. But have an architecture that would allow external LAN access, tying BMCs in different qemu instances together, perhaps serial over IPMI, and other things of that nature. -corey On 05/07/2012 10:21 AM, Anthony Liguori wrote: On 05/07/2012 10:11 AM, Avi Kivity wrote: On 05/07/2012 05:55 PM, Anthony Liguori wrote: For all intents and purposes, the BMC/RSA is a separate physical machine. That's true for any other card on a machine. It has a separate power source for all intents and purposes. If you think of it in QOM terms, what connects the nodes together ultimately is the "Vcc" pin that travels across all devices. The RTC is a little special because it has a battery backed CMOS/clock but it's also handled specially. And we fail to emulate it correctly as well, wrt. alarms. The BMC does not share Vcc. It's no different than a separate physical box. It just shares a couple buses. It controls the main power place, reset line, can read VGA and emulate keyboard, seems pretty well integrated. Emulating the keyboard is done through USB. How the VGA thing works is very vendor dependent. The VGA snooping can happen as part of the display path (essentially connected via a VGA cable) or it can be a side-band using a special graphics adapter. I think QEMU VNC emulation is a pretty good analogy actually. That is one way to do it. Figure out the interactions between two different parts in a machine, define an interface for them to communicate, and split them into two processes. We don't usually do that; I believe your motivation is that the two have different power domains (but then so do NICs with wake-on-LAN support). The power still comes from the PCI bus. Maybe. But it's on when the rest of the machine is off. So Vcc is not shared. That's all plumbed through the PCI bus FWIW. Think of something like a blade center. Each individual blade does not have it's own BMC. There's a single common BMC that provides an IPMI interface for all blades. Yet each blade still sees an IPMI interface via a USB rndis device. You can rip out the memory, PCI devices, etc. from a box while the Power is in and the BMC will be unaffected. At any rate, you would have some sort of virtual hardware device that essentially spoke QMP to the slave instance. You could just do virtio-serial and call it a day actually. Sorry I lost you. Which is the master and which is the slave? The BMC is the master, system being controlled is the slave. Ah okay. It also has to read the VGA output (say via vnc) and supply keyboard input (via sendkey). Right, QMP + VNC is a pretty accurate analogy. It really boils down to what you are trying to do. If you want to just get some piece of software working that expects to do IPMI, the easiest thing to do is run IPMI in the host and use a USB rndis interface to interact with it. That would be most strange. A remote client connecting to the IPMI interface would control the power level of the host, not the guest. IPMI with a custom backend is what I mean. That's what I mean by an IPMI -> libvirt bridge. You could build a libvirt client that exposes an IPMI interface and when you issue IPMI commands, it translate
Adding an IPMI BMC device to KVM
I've been working on adding an IPMI BMC as a virtual device under KVM. I'm doing this for two primary reasons, one to have a better test environment than what I have now for testing IPMI issues, and second to be able to better simulate a legacy environment for customers porting legacy software. For those that don't know, IPMI is a system management interface. Generally systems with IPMI have a small microcontroller, called a BMC, that is always on when the board is powered. The BMC is capable of controlling power and reset on the board, and it is hooked to sensors on the board (voltage, current, temperature, the presence of things like DIMMS and power supplies, intrusion detection, and a host of other things). The main processor on a system can communicate with the BMC over a device. Often these systems also have a LAN interface that lets you control the system remotely even when it's off. In addition, IPMI provides access to FRU (Field Replaceable Unit) data that describes the components of the system that can be replaced. It also has data records that describe the sensor, so it is possible to directly interpret the sensor data and know what the sensor is measuring without outside data. I've been struggling a bit with how to implement this. There is a lot of configuration information, and you need ways to simulate the sensors. This type of interface is a little sensitive, since it has direct access to the reset and power control of a system. I was at first considering having the BMC be an external program that KVM connected to over a chardev, with possibly a simulated LAN interface. This has the advantage that the BMC can run even when KVM is down. It could even start up KVM for a "power up", though I'm not sure how valuable that would be. Plus it could be used for other virtual machines. However, that means there is an interface to KVM over a chardev that could do nasty things, and even be a possible intrusion point. It also means there is a separate program to maintain. You could also include the BMC inside of KVM and run it as a separate thread. That way there doesn't have to be an insecure interface. But the BMC will need a lot of configuration data and this will add a bunch of code to KVM that's only tangentially related to it. And you would still need a way to simulate setting sensors and such for testing things. Either way, is this interesting for including into KVM? Does anyone have any opinions on the possible ways to implement this? Thanks, -corey -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2 v3] kvm: notify host when guest panicked
Look at drivers/char/ipmi/ipmi_msghandler.c. It has code to send panic event over IMPI. The code is pretty complex. Of course if we a going to implement something more complex than simple hypercall for panic notification we better do something more interesting with it than just saying "panic happened", like sending stack traces on all cpus for instance. I doubt that's the best example, unfortunately. The IPMI event log has limited space and it has to be send a little piece at a time since each log entry is 14 bytes. It just prints the panic string, nothing else. Not that it isn't useful, it has saved my butt before. You have lots of interesting options with paravirtualization. You could, for instance, create a console driver that delivered all console output efficiently through a hypercall. That would be really easy. Or, as you mention, a custom way to deliver panic information. Collecting information like stack traces would be harder to accomplish, as I don't think there is currently a way to get it except by sending it to printk. -corey -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
