Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Gregory Haskins wrote: > Hi Linus, > > Please pull AlacrityVM guest support for 2.6.33 from: > > git://git.kernel.org/pub/scm/linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git > for-linus > > All of these patches have stewed in linux-next for quite a while now: > > Gregory Haskins (26): I think it would be fair to point out that these patches have been objected to by the KVM folks quite extensively, on multiple technical grounds - as basically this tree forks the KVM driver space for which no valid technical reason could be offered by you in a 100+ mails long discussion. (And yes, i've been Cc:-ed to much of that thread.) The result will IMO be pain for users because now we'll have two frameworks, tooling incompatibilities, etc. etc. I've extended the Cc: for the KVM folks to have a chance to reply. Please try _much_ harder to work with the KVM folks instead of ignoring their feedback and de-facto forking their project. (and not mentioning any of this in your pull request) We should unify, not fracture. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/18/09 4:51 PM, Ingo Molnar wrote: > > * Gregory Haskins wrote: > >> Hi Linus, >> >> Please pull AlacrityVM guest support for 2.6.33 from: >> >> git://git.kernel.org/pub/scm/linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git >> for-linus >> >> All of these patches have stewed in linux-next for quite a while now: >> >> Gregory Haskins (26): > > I think it would be fair to point out that these patches have been objected > to > by the KVM folks quite extensively, Actually, these patches have nothing to do with the KVM folks. You are perhaps confusing this with the hypervisor-side discussion, of which there is indeed much disagreement. To that point, it's certainly fair to point out the controversy on the host side. It ultimately is what forced the creation of the AlacrityVM project, after all. However, it should also be pointed out that this pull request is not KVM specific, nor even KVM related per se. These patches can (and in fact, do) work in other environments that do not use KVM nor even AlacrityVM at all. VBUS, the underlying technology here, is a framework for creating optimized software-based device models using a Linux-kernel as a host and their corresponding "driver" resources to the backend. AlacrityVM is the application of these technologies using KVM/Linux/Qemu as a base, but that is an implementation detail. For more details, please see the project wiki http://developer.novell.com/wiki/index.php/AlacrityVM This pull request is for drivers to support running a Linux kernel as a guest in this environment, so it actually doesn't affect KVM in any way. They are just standard Linux drivers and in fact can load as stand-alone KMPs in any modern vanilla distro. I haven't even pushed the host side code to linux-next yet specifically because of that controversy you mention. > on multiple technical grounds - as > basically this tree forks the KVM driver space for which no valid technical > reason could be offered by you in a 100+ mails long discussion. You will have to be more specific on these technical grounds you mention, because I believe I satisfactorily rebutted any issues raised. To say that there is no technical reason is, at best, a matter of opinion. I have in fact listed numerous reasons on a technical, feature, and architectural basis on what differentiates my approach, and provided numbers which highlights their merits. Given that they are all recorded in the archives of said 100+ email thread as well as numerous others, I wont rehash the entire list here. Instead, I will post a summary of the problem space from the performance perspective, since that seems to be of most interest atm. From my research, the reason why virt in general, and KVM in particular suffers on the IO performance front is as follows: IOs (traps+interrupts) are more expensive than bare-metal, and real hardware is naturally concurrent (your hbas and nics are effectively parallel execution engines, etc). Assuming my observations are correct, in order to squeeze maximum performance from a given guest, you need to do three things: A) eliminate as many IOs as you possibly can, B) reduce the cost of the ones you can't avoid, and C) run your algorithms in parallel to emulate concurrent silicon. So to that front, we move the device models to the kernel (where they are closest to the physical IO devices) and use "cheap" instructions like PIOs/Hypercalls for (B), and exploit spare host-side SMP resources via kthreads for (C). For (A), part of the problem is that virtio-pci is not designed optimally to address the problem space, and part of it is a limitation of the PCI transport underneath it. For example, PCI is somewhat of a unique bus design in that it wants to map signals to interrupts 1:1. This works fine for real hardware where interrupts are relatively cheap, but is quite suboptimal on virt where the window-exits, injection-exits, and MMIO-based EIOs hurt substantially (multiple microseconds per). One core observation is that we don't technically need 1:1 interrupts to signals in order to function properly. Ideally we will only bother the CPU when work of a higher priority becomes ready. So the alacrityvm connector to vbus uses a model were we deploy a lockless shared-memory queue to inject interrupts. This means that temporal interrupts (of both intra and inter device variety) of similar priority can queue without incurring any extra IO. This means fewer exits, fewer EOIs, etc. The end result is that I can demonstrate that even with a single stream to a single device, I can reduce exit rate by over 45% and interrupt rate > 50% when compared to the equivalent virtio-pci ABI. This scales even higher when you add additional devices to the mix. The bottom line is that we use significantly less CPU while producing the highest throughput and lowest latency. In fact, to my knowledge vbus+venet is still the highest performing 802.x device for KVM to my knowledge, even when turning o
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/2009 05:34 PM, Gregory Haskins wrote: I think it would be fair to point out that these patches have been objected to by the KVM folks quite extensively, Actually, these patches have nothing to do with the KVM folks. You are perhaps confusing this with the hypervisor-side discussion, of which there is indeed much disagreement. This is true, though these drivers are fairly pointless for virtualization without the host side support. I did have a few issues with the guest drivers: - the duplication of effort wrt virtio. These drivers don't cover exactly the same problem space, but nearly so. - no effort at scalability - all interrupts are taken on one cpu - the patches introduce a new virtual interrupt controller for dubious (IMO) benefits From my research, the reason why virt in general, and KVM in particular suffers on the IO performance front is as follows: IOs (traps+interrupts) are more expensive than bare-metal, and real hardware is naturally concurrent (your hbas and nics are effectively parallel execution engines, etc). Assuming my observations are correct, in order to squeeze maximum performance from a given guest, you need to do three things: A) eliminate as many IOs as you possibly can, B) reduce the cost of the ones you can't avoid, and C) run your algorithms in parallel to emulate concurrent silicon. All these are addressed by vhost-net without introducing new drivers. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/09 10:43 AM, Avi Kivity wrote: > On 12/21/2009 05:34 PM, Gregory Haskins wrote: >> >>> I think it would be fair to point out that these patches have been >>> objected to >>> by the KVM folks quite extensively, >>> >> Actually, these patches have nothing to do with the KVM folks. You are >> perhaps confusing this with the hypervisor-side discussion, of which >> there is indeed much disagreement. >> > > This is true, though these drivers are fairly pointless for > virtualization without the host side support. The host side support is available in various forms (git tree, rpm, etc) from our project page. I would encourage any interested parties to check it out: Here is the git tree http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=summary Here are some RPMs: http://download.opensuse.org/repositories/devel://LLDC://alacrity/openSUSE_11.1/ And the main project site: http://developer.novell.com/wiki/index.php/AlacrityVM > > I did have a few issues with the guest drivers: > - the duplication of effort wrt virtio. These drivers don't cover > exactly the same problem space, but nearly so. Virtio itself is more or less compatible with this effort, as we have discussed (see my virtio-vbus transport, for instance). I have issues with some of the design decisions in the virtio device and ring models, but they are minor in comparison to the beef I have with the virtio-pci transport as a whole. > - no effort at scalability - all interrupts are taken on one cpu Addressed by the virtual-interrupt controller. This will enable us to route shm-signal messages to a core, under guidance from the standard irq-balance facilities. > - the patches introduce a new virtual interrupt controller for dubious > (IMO) benefits See above. Its not fully plumbed yet, which is perhaps the reason for the confusion as to its merits. Eventually I will trap the affinity calls and pass them to the host, too. Today, it at least lets us see the shm-signal statistics under /proc/interrupts, which is nice and is consistent with other IO mechanisms. > >> From my research, the reason why virt in general, and KVM in particular >> suffers on the IO performance front is as follows: IOs >> (traps+interrupts) are more expensive than bare-metal, and real hardware >> is naturally concurrent (your hbas and nics are effectively parallel >> execution engines, etc). >> >> Assuming my observations are correct, in order to squeeze maximum >> performance from a given guest, you need to do three things: A) >> eliminate as many IOs as you possibly can, B) reduce the cost of the >> ones you can't avoid, and C) run your algorithms in parallel to emulate >> concurrent silicon. >> > > All these are addressed by vhost-net without introducing new drivers. No, B and C definitely are, but A is lacking. And the performance suffers as a result in my testing (vhost-net still throws a ton of exits as its limited by virtio-pci and only adds about 1Gb/s to virtio-u, far behind venet even with things like zero-copy turned off). I will also point out that these performance aspects are only a subset of the discussion, since we are also addressing things like qos/priority, alternate fabric types, etc. I do not expect you to understand and agree where I am going per se. We can have that discussion when I once again ask you for merge consideration. But if you say "they are the same" I will call you on it, because they are demonstrably unique capability sets. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/2009 10:04 AM, Gregory Haskins wrote: No, B and C definitely are, but A is lacking. And the performance suffers as a result in my testing (vhost-net still throws a ton of exits as its limited by virtio-pci and only adds about 1Gb/s to virtio-u, far behind venet even with things like zero-copy turned off). How does virtio-pci limit vhost-net? The only time exits should occur are when the guest notifies the host that something has been placed on the ring. Since vhost-net has no tx mitigation scheme right now, the result may be that it's taking an io exit on every single packet but this is orthogonal to virtio-pci. Since virtio-pci supports MSI-X, there should be no IO exits on host->guest notification other than EOI in the virtual APIC. This is a light weight exit today and will likely disappear entirely with newer hardware. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/2009 06:37 PM, Anthony Liguori wrote: Since virtio-pci supports MSI-X, there should be no IO exits on host->guest notification other than EOI in the virtual APIC. This is a light weight exit today and will likely disappear entirely with newer hardware. I'm working on disappearing EOI exits on older hardware as well. Same idea as the old TPR patching, without most of the magic. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/09 11:37 AM, Anthony Liguori wrote: > On 12/21/2009 10:04 AM, Gregory Haskins wrote: >> No, B and C definitely are, but A is lacking. And the performance >> suffers as a result in my testing (vhost-net still throws a ton of exits >> as its limited by virtio-pci and only adds about 1Gb/s to virtio-u, far >> behind venet even with things like zero-copy turned off). >> > > How does virtio-pci limit vhost-net? The only time exits should occur > are when the guest notifies the host that something has been placed on > the ring. Since vhost-net has no tx mitigation scheme right now, the > result may be that it's taking an io exit on every single packet but > this is orthogonal to virtio-pci. > > Since virtio-pci supports MSI-X, there should be no IO exits on > host->guest notification other than EOI in the virtual APIC. The very best you can hope to achieve is 1:1 EOI per signal (though today virtio-pci is even worse than that). As I indicated above, I can eliminate more than 50% of even the EOIs in trivial examples, and even more as we scale up the number of devices or the IO load (or both). > This is a > light weight exit today and will likely disappear entirely with newer > hardware. By that argument, this is all moot. New hardware will likely obsolete the need for venet or virtio-net anyway. The goal of my work is to provide an easy to use framework for maximizing the IO transport _in lieu_ of hardware acceleration. Software will always be leading here, so we don't want to get into a pattern of waiting for new hardware to cover poor software engineering. Its simply not necessary in most cases. A little smart software design and a framework that allows it to be easily exploited/reused is the best step forward, IMO. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/09 11:40 AM, Avi Kivity wrote: > On 12/21/2009 06:37 PM, Anthony Liguori wrote: >> Since virtio-pci supports MSI-X, there should be no IO exits on >> host->guest notification other than EOI in the virtual APIC. This is >> a light weight exit today and will likely disappear entirely with >> newer hardware. > > I'm working on disappearing EOI exits on older hardware as well. Same > idea as the old TPR patching, without most of the magic. > While I applaud any engineering effort that results in more optimal execution, if you are talking about what we have discussed in the past its not quite in the same league as my proposal. You are talking about the ability to optimize the final EOI if there are no pending interrupts remaining, right? The problem with this approach is it addresses the wrong side of the curve: That is, it optimizes the code as its about to go io-idle. You still have to take an extra exit for each injection during the heat of battle, which is when you actually need it most. To that front, what I have done is re-used the lockless shared-memory concept for even "interrupt injection". Lockless shared-memory rings have the property that both producer and consumer can simultaneously manipulate the ring. So what we do in alacrityvm is deliver shm-signals (shm-signal == "interrupt" in vbus) over a ring so that the host can inject a signal to a running vcpu and a vcpu can complete an ack/re-inject cycle directly from vcpu context. Therefore, we only need a physical IDT injection when the vcpu is io-idle transitioning to io-busy, and remain completely in parallel guest/host context until we go idle again. That said, your suggestion would play nicely with the above mentioned scheme, so I look forward to seeing it in the tree. Feel free to send me patches for testing. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/2009 06:56 PM, Gregory Haskins wrote: I'm working on disappearing EOI exits on older hardware as well. Same idea as the old TPR patching, without most of the magic. While I applaud any engineering effort that results in more optimal execution, if you are talking about what we have discussed in the past its not quite in the same league as my proposal. I don't doubt this for a minute. You are talking about the ability to optimize the final EOI if there are no pending interrupts remaining, right? The problem with this approach is it addresses the wrong side of the curve: That is, it optimizes the code as its about to go io-idle. You still have to take an extra exit for each injection during the heat of battle, which is when you actually need it most. No, it's completely orthogonal. An interrupt is injected, the handler disables further interrupts and EOIs, then schedules the rest of the handling code. So long as there as packets in the ring interrupts won't be enabled and hence there won't be any reinjections. Different interrupt sources still need different interrupts, but as all of your tests have been single-interface, this can't be the reason for your performance. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/2009 10:46 AM, Gregory Haskins wrote: The very best you can hope to achieve is 1:1 EOI per signal (though today virtio-pci is even worse than that). As I indicated above, I can eliminate more than 50% of even the EOIs in trivial examples, and even more as we scale up the number of devices or the IO load (or both). If optimizing EOI is the main technical advantage of vbus, then surely we could paravirtualize EOI access and get that benefit in KVM without introducing a whole new infrastructure, no? This is a light weight exit today and will likely disappear entirely with newer hardware. By that argument, this is all moot. New hardware will likely obsolete the need for venet or virtio-net anyway. Not at all. But let's focus on concrete data. For a given workload, how many exits do you see due to EOI? They should be relatively rare because obtaining good receive batching is pretty easy. Considering these are lightweight exits (on the order of 1-2us), you need an awfully large amount of interrupts before you get really significant performance impact. You would think NAPI would kick in at this point anyway. Do you have data demonstrating the advantage of EOI mitigation? Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/09 12:05 PM, Avi Kivity wrote: > On 12/21/2009 06:56 PM, Gregory Haskins wrote: >>> I'm working on disappearing EOI exits on older hardware as well. Same >>> idea as the old TPR patching, without most of the magic. >>> >>> >> While I applaud any engineering effort that results in more optimal >> execution, if you are talking about what we have discussed in the past >> its not quite in the same league as my proposal. >> > > I don't doubt this for a minute. > >> You are talking about the ability to optimize the final EOI if there are >> no pending interrupts remaining, right? The problem with this approach >> is it addresses the wrong side of the curve: That is, it optimizes the >> code as its about to go io-idle. You still have to take an extra exit >> for each injection during the heat of battle, which is when you actually >> need it most. >> > > No, it's completely orthogonal. An interrupt is injected, the handler > disables further interrupts and EOIs, then schedules the rest of the > handling code. So long as there as packets in the ring interrupts won't > be enabled and hence there won't be any reinjections. I meant inter-vector "next-interrupt" injects. For lack of a better term, I called it reinject, but I realize in retrospect that this is ambiguous. > > Different interrupt sources still need different interrupts, but as all > of your tests have been single-interface, this can't be the reason for > your performance. > Actually I have tested both single and multi-homed setups, but it doesn't matter. Even a single device can benefit, as even single devices may have multiple vector sources that are highly probably to generate coincident events. For instance, consider that even a basic ethernet may have separate vectors for "rx" and "tx-complete". A simple ping is likely to generate both vectors at approximately the same time, given how the host side resources often work. Trying to condense multiple vectors into one means its up to the driver to implement any type of prioritization on its own (or worse, it just suffers from PI). Likewise, implementing them as unique vectors means you are likely to have coincident events for certain workloads. What alacrityvm tries to do is recognize these points and optimize for both cases. It means we still retain framework-managed prioritized callbacks, yet optimize away extraneous IO for coincident signals. IOW: best of both worlds. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/09 12:20 PM, Anthony Liguori wrote: > On 12/21/2009 10:46 AM, Gregory Haskins wrote: >> The very best you can hope to achieve is 1:1 EOI per signal (though >> today virtio-pci is even worse than that). As I indicated above, I can >> eliminate more than 50% of even the EOIs in trivial examples, and even >> more as we scale up the number of devices or the IO load (or both). >> > > If optimizing EOI is the main technical advantage of vbus, then surely > we could paravirtualize EOI access and get that benefit in KVM without > introducing a whole new infrastructure, no? No, because I never claimed optimizing EOI was the main/only advantage. The feature set has all been covered in extensive detail in the lists, however, so I will defer you to google for the archives for your reading pleasure. > >>> This is a >>> light weight exit today and will likely disappear entirely with newer >>> hardware. >>> >> By that argument, this is all moot. New hardware will likely obsolete >> the need for venet or virtio-net anyway. > > Not at all. Well, surely something like SR-IOV is moving in that direction, no? > But let's focus on concrete data. For a given workload, > how many exits do you see due to EOI? Its of course highly workload dependent, and I've published these details in the past, I believe. Off the top of my head, I recall that virtio-pci tends to throw about 65k exits per second, vs about 32k/s for venet on a 10GE box, but I don't recall what ratio of those exits are EOI. To be perfectly honest, I don't care. I do not discriminate against the exit type...I want to eliminate as many as possible, regardless of the type. That's how you go fast and yet use less CPU. > They should be relatively rare > because obtaining good receive batching is pretty easy. Batching is poor mans throughput (its easy when you dont care about latency), so we generally avoid as much as possible. > Considering > these are lightweight exits (on the order of 1-2us), APIC EOIs on x86 are MMIO based, so they are generally much heavier than that. I measure at least 4-5us just for the MMIO exit on my Woodcrest, never mind executing the locking/apic-emulation code. > you need an awfully > large amount of interrupts before you get really significant performance > impact. You would think NAPI would kick in at this point anyway. > Whether NAPI can kick in or not is workload dependent, and it also does not address coincident events. But on that topic, you can think of AlacrityVM's interrupt controller as "NAPI for interrupts", because it operates on the same principle. For what its worth, it also operates on a "NAPI for hypercalls" concept too. > Do you have data demonstrating the advantage of EOI mitigation? I have non-scientifically gathered numbers in my notebook that put it on average of about 55%-60% reduction in EOIs for inbound netperf runs, for instance. I don't have time to gather more in the near term, but its typically in that range for a chatty enough workload, and it goes up as you add devices. I would certainly formally generate those numbers when I make another merge request in the future, but I don't have them now. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/2009 11:44 AM, Gregory Haskins wrote: Well, surely something like SR-IOV is moving in that direction, no? Not really, but that's a different discussion. But let's focus on concrete data. For a given workload, how many exits do you see due to EOI? Its of course highly workload dependent, and I've published these details in the past, I believe. Off the top of my head, I recall that virtio-pci tends to throw about 65k exits per second, vs about 32k/s for venet on a 10GE box, but I don't recall what ratio of those exits are EOI. Was this userspace virtio-pci or was this vhost-net? If it was the former, then were you using MSI-X? If you weren't, there would be an additional (rather heavy) exit per-interrupt to clear the ISR which would certainly account for a large portion of the additional exits. To be perfectly honest, I don't care. I do not discriminate against the exit type...I want to eliminate as many as possible, regardless of the type. That's how you go fast and yet use less CPU. It's important to understand why one mechanism is better than another. All I'm looking for is a set of bullet points that say, vbus does this, vhost-net does that, therefore vbus is better. We would then either say, oh, that's a good idea, let's change vhost-net to do that, or we would say, hrm, well, we can't change vhost-net to do that because of some fundamental flaw, let's drop it and adopt vbus. It's really that simple :-) They should be relatively rare because obtaining good receive batching is pretty easy. Batching is poor mans throughput (its easy when you dont care about latency), so we generally avoid as much as possible. Fair enough. Considering these are lightweight exits (on the order of 1-2us), APIC EOIs on x86 are MMIO based, so they are generally much heavier than that. I measure at least 4-5us just for the MMIO exit on my Woodcrest, never mind executing the locking/apic-emulation code. You won't like to hear me say this, but Woodcrests are pretty old and clunky as far as VT goes :-) On a modern Nehalem, I would be surprised if an MMIO exit handled in the kernel was muck more than 2us. The hardware is getting very, very fast. The trends here are very important to consider when we're looking at architectures that we potentially are going to support for a long time. you need an awfully large amount of interrupts before you get really significant performance impact. You would think NAPI would kick in at this point anyway. Whether NAPI can kick in or not is workload dependent, and it also does not address coincident events. But on that topic, you can think of AlacrityVM's interrupt controller as "NAPI for interrupts", because it operates on the same principle. For what its worth, it also operates on a "NAPI for hypercalls" concept too. The concept of always batching hypercalls has certainly been explored within the context of Xen. But then when you look at something like KVM's hypercall support, it turns out that with sufficient cleverness in the host, we don't even bother with the MMU hypercalls anymore. Doing fancy things in the guest is difficult to support from a long term perspective. It'll more or less never work for Windows and even the lag with Linux makes it difficult for users to see the benefit of these changes. You get a lot more flexibility trying to solve things in the host even if it's convoluted (like TPR patching). Do you have data demonstrating the advantage of EOI mitigation? I have non-scientifically gathered numbers in my notebook that put it on average of about 55%-60% reduction in EOIs for inbound netperf runs, for instance. I don't have time to gather more in the near term, but its typically in that range for a chatty enough workload, and it goes up as you add devices. I would certainly formally generate those numbers when I make another merge request in the future, but I don't have them now. I don't think it's possible to make progress with vbus without detailed performance data comparing both vbus and virtio (vhost-net). On the virtio/vhost-net side, I think we'd be glad to help gather/analyze that data. We have to understand why one's better than the other and then we have to evaluate whether we can bring those benefits into the later. If we can't, we merge vbus. If we can, we fix virtio. Regards, Anthony Liguori Kind Regards, -Greg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Mon, Dec 21, 2009 at 12:44:17PM -0500, Gregory Haskins wrote: > > They should be relatively rare > > because obtaining good receive batching is pretty easy. > > Batching is poor mans throughput (its easy when you dont care about > latency), so we generally avoid as much as possible. > > > Considering > > these are lightweight exits (on the order of 1-2us), > > APIC EOIs on x86 are MMIO based, so they are generally much heavier than > that. I measure at least 4-5us just for the MMIO exit on my Woodcrest, > never mind executing the locking/apic-emulation code. > With x2apic EOIs are not MMIO any longer. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Gregory Haskins wrote: > On 12/18/09 4:51 PM, Ingo Molnar wrote: > > > > * Gregory Haskins wrote: > > > >> Hi Linus, > >> > >> Please pull AlacrityVM guest support for 2.6.33 from: > >> > >> git://git.kernel.org/pub/scm/linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git > >> for-linus > >> > >> All of these patches have stewed in linux-next for quite a while now: > >> > >> Gregory Haskins (26): > > > > I think it would be fair to point out that these patches have been objected > > to > > by the KVM folks quite extensively, > > Actually, these patches have nothing to do with the KVM folks. [...] That claim is curious to me - the AlacrityVM host is 90% based on KVM code, so how can it not be about KVM? I just checked, most of the changes that AlacrityVM host does to KVM is in adding the host side interfaces for these guest drivers: virt/kvm/Kconfig | 11 + virt/kvm/coalesced_mmio.c | 65 +++--- virt/kvm/coalesced_mmio.h |1 + virt/kvm/eventfd.c| 599 + virt/kvm/ioapic.c | 118 +++-- virt/kvm/ioapic.h |5 + virt/kvm/iodev.h | 55 +++-- virt/kvm/irq_comm.c | 267 ++--- virt/kvm/kvm_main.c | 127 -- virt/kvm/xinterface.c | 587 10 files changed, 1649 insertions(+), 186 deletions(-) [ stat for virt/kvm/ taken as of today, AlacrityVM host tree commit 84afcc7 ] So as far as kernel code modifications of AlacrityVM goes, it's very much about KVM. > [...] You are perhaps confusing this with the hypervisor-side discussion, > of which there is indeed much disagreement. Are the guest drivers living in a vacuum? The whole purpose of the AlacrityVM guest drivers is to ... enable AlacrityVM support, right? So how can it be not about KVM? Gregory, it would be nice if you worked _much_ harder with the KVM folks before giving up. It's not like there's much valid technical disagreement that i can identify in any of the threads - the strongest one i could identify was: "I want to fork KVM so please let me do it, nobody is harmed, choice is good". Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Anthony Liguori wrote: > It's important to understand why one mechanism is better than another. All > I'm looking for is a set of bullet points that say, vbus does this, > vhost-net does that, therefore vbus is better. We would then either say, > oh, that's a good idea, let's change vhost-net to do that, or we would say, > hrm, well, we can't change vhost-net to do that because of some fundamental > flaw, let's drop it and adopt vbus. > > It's really that simple :-) That makes a lot of sense to me. I think we better have damn good technical reasons before we encourage a fork of a subsystem within the kernel. Technical truth is not something we can 'agree to disagree' on, and it is not something we can compromise on really. Both the host and the guest code is in Linux so adding another variant without that variant replacing the old one (on the spot or gradually) makes no technical sense. Gregory, i'd suggest for you to shape this as a "this and this aspect of KVM needs to be replaced/fixed" list of items, as suggested by Anthony. In my experience the KVM folks are very approachable and very reasonable about addressing technical shortcomings and acting upon feedback (and are happily accepting code as well) - so to the extent there's room for improvement here it should be done by shaping KVM, not by forking and rebranding it. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
Anthony Liguori writes: > > On a modern Nehalem, I would be surprised if an MMIO exit handled in > the kernel was muck more than 2us. The hardware is getting very, very > fast. The trends here are very important to consider when we're > looking at architectures that we potentially are going to support for > a long time. When you talk about trends the trend for IO is also to get faster. An exit will be always more expensive than passing something from another CPU in shared memory. An exit is much more work, with lots of saved context and fundamentally synchronous, even with all the tricks hardware can do. And then there's the in kernel handler too. Shared memory passing from another CPU is a much cheaper operation and more likely to scale with IO rate improvements. The basic problem in this discussion seems to be the usual disconnect between working code (I understand Gregory has working code that demonstrates the performance advances he's claiming) versus unwritten optimizations. Unwritten code tends to always sound nicer, but it remains to be seen if it can deliver what it promises. >From a abstract stand point having efficient paravirtual IO interfaces seem attractive. I also personally don't see a big problem in having another set of virtual drivers -- Linux already has plenty (vmware, xen, virtio, power, s390-vm, ...) and it's not that they would be a particular maintenance burden impacting the kernel core. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 05:49 AM, Andi Kleen wrote: Anthony Liguori writes: On a modern Nehalem, I would be surprised if an MMIO exit handled in the kernel was muck more than 2us. The hardware is getting very, very fast. The trends here are very important to consider when we're looking at architectures that we potentially are going to support for a long time. When you talk about trends the trend for IO is also to get faster. An exit will be always more expensive than passing something from another CPU in shared memory. An exit is much more work, with lots of saved context and fundamentally synchronous, even with all the tricks hardware can do. And then there's the in kernel handler too. Noone is advocating avoiding shared memory and doing more exits in the IO path :-) Whether it's x2apic support or whether it's more sophisicated hardware apic virtualization support, the point remains that taking an exit due to EOI is likely to not be required in the near term future. So far, the only actual technical advantage I've seen is that vbus avoids EOI exits. My response is that I don't think that that's so important especially when you consider that it's going to not matter so much in the future and that Avi has some ideas about how to eliminate some of them even on older hardware. I'm also suspicious that EOI exits alone would result in a huge performance differiental between the two architectures. We think we understand why vbus does better than the current userspace virtio backend. That's why we're building vhost-net. It's not done yet, but our expectation is that it will do just as well if not better. Shared memory passing from another CPU is a much cheaper operation and more likely to scale with IO rate improvements. The basic problem in this discussion seems to be the usual disconnect between working code (I understand Gregory has working code that demonstrates the performance advances he's claiming) versus unwritten optimizations. vbus has one driver (networking) that supports one guest (very new Linux kernels). It supports one hypervisor (KVM) on one architecture (x86). On the other hand, virtio has six upstream drivers (console, network, block, rng, balloon, 9p) with at least as many in development. It supports kernels going back to at least 2.6.18, almost all versions of Windows, and has experimental drivers for other OSes. It supports KVM, lguest, VirtualBox, with support for additional hypervisors under development. It supports at least five architectures (x86, ppc, s390, ia64, arm). You are correct, vbus has better numbers than virtio today. But so far, it's hardly an apples-to-apples comparison. Our backend networking driver has been implemented entirely in userspace up until very recently. There really isn't any good performance data comparing vbus to vhost-net largely because vhost-net is still under active development. The most important point though, is that so far, I don't think Greg has been able to articulate _why_ vbus would perform better than vhost-net. If that can be articulated in a way that we all agree vbus has a technical advantage over vhost-net, then I'm absolutely in agreement that it should be merged. I think the comparison would be if someone submitted a second e1000 driver that happened to do better on one netperf test than the current e1000 driver. You can argue, hey, choice is good, let's let a user choose if they want to use the faster e1000 driver. But surely, the best thing for a user is to figure out why the second e1000 driver is better on that one test, integrate that change into the current e1000 driver, or decided that the new e1000 driver is more superior in architecture and do the required work to make the new e1000 driver a full replacement for the old one. Regards, Anthony Liguori Unwritten code tends to always sound nicer, but it remains to be seen if it can deliver what it promises. From a abstract stand point having efficient paravirtual IO interfaces seem attractive. I also personally don't see a big problem in having another set of virtual drivers -- Linux already has plenty (vmware, xen, virtio, power, s390-vm, ...) and it's not that they would be a particular maintenance burden impacting the kernel core. -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Tuesday 22 December 2009 04:31:32 pm Anthony Liguori wrote: > I think the comparison would be if someone submitted a second e1000 > driver that happened to do better on one netperf test than the current > e1000 driver. > > You can argue, hey, choice is good, let's let a user choose if they want > to use the faster e1000 driver. But surely, the best thing for a user > is to figure out why the second e1000 driver is better on that one test, > integrate that change into the current e1000 driver, or decided that the Even though this is "Won't somebody please think of the users?" argument such work would be much welcomed. Sending patches would be a great start.. > new e1000 driver is more superior in architecture and do the required > work to make the new e1000 driver a full replacement for the old one. Right, like everyone actually does things this way.. I wonder why do we have OSS, old Firewire and IDE stacks still around then? > Regards, > > Anthony Liguori > > > Unwritten code tends to always sound nicer, but it remains to be seen > > if it can deliver what it promises. > > > > From a abstract stand point having efficient paravirtual IO interfaces > > seem attractive. > > > > I also personally don't see a big problem in having another set of > > virtual drivers -- Linux already has plenty (vmware, xen, virtio, power, > > s390-vm, ...) and it's not that they would be a particular maintenance > > burden impacting the kernel core. Exactly, I also don't see any problem here, especially since AlacrityVM drivers have much cleaner design / internal architecture than some of their competitors.. -- Bartlomiej Zolnierkiewicz -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> So far, the only actual technical advantage I've seen is that vbus avoids > EOI exits. The technical advantage is that it's significantly faster today. Maybe your proposed alternative is as fast, or maybe it's not. Who knows? > We think we understand why vbus does better than the current userspace > virtio backend. That's why we're building vhost-net. It's not done yet, > but our expectation is that it will do just as well if not better. That's the vapourware vs working code disconnect I mentioned. One side has hard numbers&working code and the other has expectations. I usually find it sad when the vapourware holds up the working code. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 10:01 AM, Bartlomiej Zolnierkiewicz wrote: new e1000 driver is more superior in architecture and do the required work to make the new e1000 driver a full replacement for the old one. Right, like everyone actually does things this way.. I wonder why do we have OSS, old Firewire and IDE stacks still around then? And it's always a source of pain, isn't it. I also personally don't see a big problem in having another set of virtual drivers -- Linux already has plenty (vmware, xen, virtio, power, s390-vm, ...) and it's not that they would be a particular maintenance burden impacting the kernel core. Exactly, I also don't see any problem here, especially since AlacrityVM drivers have much cleaner design / internal architecture than some of their competitors.. Care to provide some actual objective argument to why it's better than what we already have? Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 10:21 AM, Andi Kleen wrote: So far, the only actual technical advantage I've seen is that vbus avoids EOI exits. The technical advantage is that it's significantly faster today. There are two separate pieces of code in question. There are front-end drivers and there are back-end drivers. Right now, there are only front-end drivers in the kernel today. The combination of vbus front-end drivers and *kernel* back-end drivers are faster than the *combination* of virtio front-end drivers and *userspace* back-end drivers. vhost-net is our kernel back-end driver. No one has yet established that the combination of virtio front-end driver and kernel back-end driver is really significantly slower than vbus. Maybe your proposed alternative is as fast, or maybe it's not. Who knows? We think we understand why vbus does better than the current userspace virtio backend. That's why we're building vhost-net. It's not done yet, but our expectation is that it will do just as well if not better. That's the vapourware vs working code disconnect I mentioned. One side has hard numbers&working code and the other has expectations. I usually find it sad when the vapourware holds up the working code. We're not talking about vaporware. vhost-net exists. Regards, Anthony Liguori -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 06:21 PM, Andi Kleen wrote: So far, the only actual technical advantage I've seen is that vbus avoids EOI exits. The technical advantage is that it's significantly faster today. Maybe your proposed alternative is as fast, or maybe it's not. Who knows? We're working on numbers for the proposed alternative, so we should know soon. Are the AlacrityVM folks working on having all the virtio drivers for all the virtio archs? We shouldn't drop everything and switch to new code just because someone came up with a new idea. The default should be to enhance the existing code. We think we understand why vbus does better than the current userspace virtio backend. That's why we're building vhost-net. It's not done yet, but our expectation is that it will do just as well if not better. That's the vapourware vs working code disconnect I mentioned. One side has hard numbers&working code and the other has expectations. I usually find it sad when the vapourware holds up the working code. vhost-net is working code and is queued for 2.6.33. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> We're not talking about vaporware. vhost-net exists. Is it as fast as the alacrityvm setup then e.g. for network traffic? Last I heard the first could do wirespeed 10Gbit/s on standard hardware. Can vhost-net do the same thing? -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 07:33 PM, Andi Kleen wrote: We're not talking about vaporware. vhost-net exists. Is it as fast as the alacrityvm setup then e.g. for network traffic? Last I heard the first could do wirespeed 10Gbit/s on standard hardware. That was with zero-copy IIRC, which is known broken. There's nothing alacrity-specific about zerocopy (and in fact the first zerocopy patches were from Rusty). Can vhost-net do the same thing? I've heard unofficial numbers which approach that, but let's wait for the official ones. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 2:57 AM, Ingo Molnar wrote: > > * Gregory Haskins wrote: > >> On 12/18/09 4:51 PM, Ingo Molnar wrote: >>> >>> * Gregory Haskins wrote: >>> Hi Linus, Please pull AlacrityVM guest support for 2.6.33 from: git://git.kernel.org/pub/scm/linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git for-linus All of these patches have stewed in linux-next for quite a while now: Gregory Haskins (26): >>> >>> I think it would be fair to point out that these patches have been objected >>> to >>> by the KVM folks quite extensively, >> >> Actually, these patches have nothing to do with the KVM folks. [...] > > That claim is curious to me - the AlacrityVM host It's quite simple, really. These drivers support accessing vbus, and vbus is hypervisor agnostic. In fact, vbus isn't necessarily even hypervisor related. It may be used anywhere where a Linux kernel is the "io backend", which includes hypervisors like AlacrityVM, but also userspace apps, and interconnected physical systems as well. The vbus-core on the backend, and the drivers on the frontend operate completely independent of the underlying hypervisor. A glue piece called a "connector" ties them together, and any "hypervisor" specific details are encapsulated in the connector module. In this case, the connector surfaces to the guest side as a pci-bridge, so even that is not hypervisor specific per se. It will work with any pci-bridge that exposes a compatible ABI, which conceivably could be actual hardware. The AlacrityVM project just so happens to be the primary consumer, and is therefore the most convenient way to package them up at the moment. > is 90% based on KVM code, so > how can it not be about KVM? I just checked, most of the changes that > AlacrityVM host does to KVM is in adding the host side interfaces for these > guest drivers: > > virt/kvm/Kconfig | 11 + > virt/kvm/coalesced_mmio.c | 65 +++--- > virt/kvm/coalesced_mmio.h |1 + > virt/kvm/eventfd.c| 599 > + > virt/kvm/ioapic.c | 118 +++-- > virt/kvm/ioapic.h |5 + > virt/kvm/iodev.h | 55 +++-- > virt/kvm/irq_comm.c | 267 ++--- > virt/kvm/kvm_main.c | 127 -- > virt/kvm/xinterface.c | 587 > 10 files changed, 1649 insertions(+), 186 deletions(-) > > [ stat for virt/kvm/ taken as of today, AlacrityVM host tree commit 84afcc7 ] > > So as far as kernel code modifications of AlacrityVM goes, it's very much > about KVM. I think you are confused. Even if we entertained the notion that the host side diffstat were somehow relevant here, you are probably comparing the kvm.git backports that are in my tree. The only real KVM specific change that is in my tree is the 587 lines for the xinterface.c module, which is about ~4%, not 90%. Also note that I have pushed this xinterface logic upstream already, but it just hasn't been accepted yet. If I wanted to be extremely generous, you could include the entire "KVM connector" code that bridges vbus-core to kvm-core, but even that tops out at a total of ~17% of the changes in my tree. So I am still not seeing the 90% nor how it is relevant. > >> [...] You are perhaps confusing this with the hypervisor-side discussion, >> of which there is indeed much disagreement. > > Are the guest drivers living in a vacuum? The whole purpose of the AlacrityVM > guest drivers is to ... enable AlacrityVM support, right? More specifically, the purpose of the drivers, like any driver, is to enable support for the underlying device in which it is related to. In this case, the devices are vbus based devices. Of those, AlacrityVM is the only available platform that exposes them. However, that is a maturity/adoption detail, not a technical limitation. Simply implementing a new connector would bridge these drivers to other environments as well. There are community members working on these as we speak, as a matter of fact. > So how can it be not about KVM? Because AlacrityVM is a hypervisor that supports VBUS for PV IO, and KVM is not. In addition, the presence of these drivers in no way alters, interferes with, or diminishes features found in KVM today. So it is, and never will be about KVM until upstream KVM decides that they want to support VBUS based PV-IO. If you want to talk about the host side, then I have +587 lines that hang in the balance that affect KVM, yes. But that isn't what $subject was about. > > Gregory, it would be nice if you worked _much_ harder with the KVM folks > before giving up. I think the 5+ months that I politely tried to convince the KVM folks that this was a good idea was pretty generous of my employer. The KVM maintainers have ultimately made it clear they are not interested in directly supporting this concept (which is their prerogative), but are perhaps will
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 07:36 PM, Gregory Haskins wrote: Gregory, it would be nice if you worked _much_ harder with the KVM folks before giving up. I think the 5+ months that I politely tried to convince the KVM folks that this was a good idea was pretty generous of my employer. The KVM maintainers have ultimately made it clear they are not interested in directly supporting this concept (which is their prerogative), but are perhaps willing to support the peripheral logic needed to allow it to easily interface with KVM. I can accept that, and thus AlacrityVM was born. Review pointed out locking issues with xinterface which I have not seen addressed. I asked why the irqfd/ioeventfd mechanisms are insufficient, and you did not reply. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 1:53 PM, Avi Kivity wrote: > On 12/22/2009 07:36 PM, Gregory Haskins wrote: >> >>> Gregory, it would be nice if you worked _much_ harder with the KVM folks >>> before giving up. >>> >> I think the 5+ months that I politely tried to convince the KVM folks >> that this was a good idea was pretty generous of my employer. The KVM >> maintainers have ultimately made it clear they are not interested in >> directly supporting this concept (which is their prerogative), but are >> perhaps willing to support the peripheral logic needed to allow it to >> easily interface with KVM. I can accept that, and thus AlacrityVM was >> born. >> > > Review pointed out locking issues with xinterface which I have not seen > addressed. I asked why the irqfd/ioeventfd mechanisms are insufficient, > and you did not reply. > Yes, I understand. I've been too busy to rework the code for an upstream push. I will certainly address those questions when I make the next attempt, but they weren't relevant to the guest side. signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 1:53 PM, Avi Kivity wrote: > I asked why the irqfd/ioeventfd mechanisms are insufficient, and you did not > reply. > BTW: the ioeventfd issue just fell through the cracks, so sorry about that. Note that I have no specific issue with irqfd ever since the lockless IRQ injection code was added. ioeventfd turned out to be suboptimal for me in the fast path for two reasons: 1) the underlying eventfd is called in atomic context. I had posted patches to Davide to address that limitation, but I believe he rejected them on the grounds that they are only relevant to KVM. 2) it cannot retain the data field passed in the PIO. I wanted to have one vector that could tell me what value was written, and this cannot be expressed in ioeventfd. Based on this, it was a better decision to add a ioevent interface to xinterface. It neatly solves both problems. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 09:15 PM, Gregory Haskins wrote: On 12/22/09 1:53 PM, Avi Kivity wrote: I asked why the irqfd/ioeventfd mechanisms are insufficient, and you did not reply. BTW: the ioeventfd issue just fell through the cracks, so sorry about that. Note that I have no specific issue with irqfd ever since the lockless IRQ injection code was added. ioeventfd turned out to be suboptimal for me in the fast path for two reasons: 1) the underlying eventfd is called in atomic context. I had posted patches to Davide to address that limitation, but I believe he rejected them on the grounds that they are only relevant to KVM. If you're not doing something pretty minor, you're better of waking up a thread (perhaps _sync if you want to keep on the same cpu). With the new user return notifier thingie, that's pretty cheap. 2) it cannot retain the data field passed in the PIO. I wanted to have one vector that could tell me what value was written, and this cannot be expressed in ioeventfd. It would be easier to add data logging support to ioeventfd, if it was needed that badly. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 2:25 PM, Avi Kivity wrote: > On 12/22/2009 09:15 PM, Gregory Haskins wrote: >> On 12/22/09 1:53 PM, Avi Kivity wrote: >> >>> I asked why the irqfd/ioeventfd mechanisms are insufficient, and you >>> did not reply. >>> >>> >> BTW: the ioeventfd issue just fell through the cracks, so sorry about >> that. Note that I have no specific issue with irqfd ever since the >> lockless IRQ injection code was added. >> >> ioeventfd turned out to be suboptimal for me in the fast path for two >> reasons: >> >> 1) the underlying eventfd is called in atomic context. I had posted >> patches to Davide to address that limitation, but I believe he rejected >> them on the grounds that they are only relevant to KVM. >> > > If you're not doing something pretty minor, you're better of waking up a > thread (perhaps _sync if you want to keep on the same cpu). With the > new user return notifier thingie, that's pretty cheap. We have exploits that take advantage of IO heuristics. When triggered they do more work in vcpu context than normal, which reduces latency under certain circumstances. But you definitely do _not_ want to do them in-atomic ;) > >> 2) it cannot retain the data field passed in the PIO. I wanted to have >> one vector that could tell me what value was written, and this cannot be >> expressed in ioeventfd. >> >> > > It would be easier to add data logging support to ioeventfd, if it was > needed that badly. "Better design"? perhaps. "More easily"? no. Besides, Davide has already expressed dissatisfaction with the KVM-isms creeping into eventfd, so its not likely to ever be accepted regardless of your own disposition. xinterface, as it turns out, is a great KVM interface for me and easy to extend, all without conflicting with the changes in upstream. The old way was via the kvm ioctl interface, but that sucked as the ABI was always moving. Where is the problem? ioeventfd still works fine as it is. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 2:32 PM, Gregory Haskins wrote: > On 12/22/09 2:25 PM, Avi Kivity wrote: >> >> If you're not doing something pretty minor, you're better of waking up a >> thread (perhaps _sync if you want to keep on the same cpu). With the >> new user return notifier thingie, that's pretty cheap. > > We have exploits that take advantage of IO heuristics. When triggered > they do more work in vcpu context than normal, which reduces latency > under certain circumstances. But you definitely do _not_ want to do > them in-atomic ;) And I almost forgot: dev->call() is an RPC to the backend device. Therefore, it must be synchronous, yet we dont want it locked either. I think that was actually the primary motivation for the change, now that I think about it. -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 09:32 PM, Gregory Haskins wrote: xinterface, as it turns out, is a great KVM interface for me and easy to extend, all without conflicting with the changes in upstream. The old way was via the kvm ioctl interface, but that sucked as the ABI was always moving. Where is the problem? ioeventfd still works fine as it is. It means that kvm locking suddenly affects more of the kernel. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 2:38 PM, Avi Kivity wrote: > On 12/22/2009 09:32 PM, Gregory Haskins wrote: >> xinterface, as it turns out, is a great KVM interface for me and easy to >> extend, all without conflicting with the changes in upstream. The old >> way was via the kvm ioctl interface, but that sucked as the ABI was >> always moving. Where is the problem? ioeventfd still works fine as >> it is. >> > > It means that kvm locking suddenly affects more of the kernel. > Thats ok. This would only be w.r.t. devices that are bound to the KVM instance anyway, so they better know what they are doing (and they do). -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 09:32 PM, Gregory Haskins wrote: Besides, Davide has already expressed dissatisfaction with the KVM-isms creeping into eventfd, so its not likely to ever be accepted regardless of your own disposition. Why don't you duplicate eventfd, then, should be easier than duplicating virtio. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 09:41 PM, Gregory Haskins wrote: It means that kvm locking suddenly affects more of the kernel. Thats ok. This would only be w.r.t. devices that are bound to the KVM instance anyway, so they better know what they are doing (and they do). It's okay to the author of that device. It's not okay to the kvm developers who are still evolving the locking and have to handle all devices that use xinterface. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 2:43 PM, Avi Kivity wrote: > On 12/22/2009 09:41 PM, Gregory Haskins wrote: >> >>> It means that kvm locking suddenly affects more of the kernel. >>> >>> >> Thats ok. This would only be w.r.t. devices that are bound to the KVM >> instance anyway, so they better know what they are doing (and they do). >> >> > > It's okay to the author of that device. It's not okay to the kvm > developers who are still evolving the locking and have to handle all > devices that use xinterface. Perhaps, but like it or not, if you want to do in-kernel you need to invoke backends. And if you want to invoke backends, limiting it to thread wakeups is, well, limiting. For one, you miss out on that exploit I mentioned earlier which can help sometimes. Besides, the direction that Marcelo and I left the mmio/pio bus was that it would go lockless eventually, not "more lockful" ;) Has that changed? I honestly haven't followed whats going on in the io-bus code in a while. -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Tue, 22 Dec 2009, Gregory Haskins wrote: > On 12/22/09 1:53 PM, Avi Kivity wrote: > > I asked why the irqfd/ioeventfd mechanisms are insufficient, and you did > > not reply. > > > > BTW: the ioeventfd issue just fell through the cracks, so sorry about > that. Note that I have no specific issue with irqfd ever since the > lockless IRQ injection code was added. > > ioeventfd turned out to be suboptimal for me in the fast path for two > reasons: > > 1) the underlying eventfd is called in atomic context. I had posted > patches to Davide to address that limitation, but I believe he rejected > them on the grounds that they are only relevant to KVM. I thought we addressed this already, in the few hundreds of email we exchanged back then :) > 2) it cannot retain the data field passed in the PIO. I wanted to have > one vector that could tell me what value was written, and this cannot be > expressed in ioeventfd. Like might have hinted in his reply, couldn't you add data support to the ioeventfd bits in KVM, instead of leaking them into mainline eventfd? - Davide -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/09 2:39 PM, Davide Libenzi wrote: > On Tue, 22 Dec 2009, Gregory Haskins wrote: > >> On 12/22/09 1:53 PM, Avi Kivity wrote: >>> I asked why the irqfd/ioeventfd mechanisms are insufficient, and you did >>> not reply. >>> >> >> BTW: the ioeventfd issue just fell through the cracks, so sorry about >> that. Note that I have no specific issue with irqfd ever since the >> lockless IRQ injection code was added. >> >> ioeventfd turned out to be suboptimal for me in the fast path for two >> reasons: >> >> 1) the underlying eventfd is called in atomic context. I had posted >> patches to Davide to address that limitation, but I believe he rejected >> them on the grounds that they are only relevant to KVM. > > I thought we addressed this already, in the few hundreds of email we > exchanged back then :) We addressed the race conditions, but not the atomic callbacks. I can't remember exactly what you said, but the effect was "no", so I dropped it. ;) This was the thread. http://www.archivum.info/linux-ker...@vger.kernel.org/2009-06/08548/Re:_[KVM-RFC_PATCH_1_2]_eventfd:_add_an_explicit_srcu_based_notifier_interface > > > >> 2) it cannot retain the data field passed in the PIO. I wanted to have >> one vector that could tell me what value was written, and this cannot be >> expressed in ioeventfd. > > Like might have hinted in his reply, couldn't you add data support to the > ioeventfd bits in KVM, instead of leaking them into mainline eventfd? > Perhaps, or even easier I could extend xinterface. Which is what I did ;) The problem with the first proposal is that you would no longer actually have an eventfd based mechanism...so any code using ioeventfd (like Michael Tsirkin's for instance) would break. Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/21/09 7:12 PM, Anthony Liguori wrote: > On 12/21/2009 11:44 AM, Gregory Haskins wrote: >> Well, surely something like SR-IOV is moving in that direction, no? >> > > Not really, but that's a different discussion. Ok, but my general point still stands. At some level, some crafty hardware engineer may invent something that obsoletes the need for, say, PV 802.x drivers because it can hit 40GE line rate at the same performance level of bare metal with some kind of pass-through trick. But I still do not see that as an excuse for sloppy software in the meantime, as there will always be older platforms, older IO cards, or different IO types that are not benefactors of said hw based optimizations. > >>> But let's focus on concrete data. For a given workload, >>> how many exits do you see due to EOI? >>> >> Its of course highly workload dependent, and I've published these >> details in the past, I believe. Off the top of my head, I recall that >> virtio-pci tends to throw about 65k exits per second, vs about 32k/s for >> venet on a 10GE box, but I don't recall what ratio of those exits are >> EOI. > > Was this userspace virtio-pci or was this vhost-net? Both, actually, though userspace is obviously even worse. > If it was the > former, then were you using MSI-X? MSI-X > If you weren't, there would be an > additional (rather heavy) exit per-interrupt to clear the ISR which > would certainly account for a large portion of the additional exits. > Yep, if you don't use MSI it is significantly worse as expected. >>To be perfectly honest, I don't care. I do not discriminate >> against the exit type...I want to eliminate as many as possible, >> regardless of the type. That's how you go fast and yet use less CPU. >> > > It's important to understand why one mechanism is better than another. Agreed, but note _I_ already understand why. I've certainly spent countless hours/emails trying to get others to understand as well, but it seems most are too busy to actually listen. > All I'm looking for is a set of bullet points that say, vbus does this, > vhost-net does that, therefore vbus is better. We would then either > say, oh, that's a good idea, let's change vhost-net to do that, or we > would say, hrm, well, we can't change vhost-net to do that because of > some fundamental flaw, let's drop it and adopt vbus. > > It's really that simple :-) This is all been covered ad-nauseam, directly with youself in many cases. Google is your friend. Here are some tips while you research: Do not fall into the trap of vhost-net vs vbus, or venet vs virtio-net, or you miss the point entirely. Recall that venet was originally crafted to demonstrate the virtues of my three performance objectives (kill exits, reduce exit overhead, and run concurrently). Then there is all the stuff we are laying on top, like qos, real-time, advanced fabrics, and easy adoption for various environments (so it doesn't need to be redefined each time). Therefore if you only look at the limited feature set of virtio-net, you will miss the majority of the points of the framework. virtio tried to capture some of these ideas, but it missed the mark on several levels and was only partially defined. Incidentally, you can stil run virtio over vbus if desired, but so far no one has tried to use my transport. > > >>> They should be relatively rare >>> because obtaining good receive batching is pretty easy. >>> >> Batching is poor mans throughput (its easy when you dont care about >> latency), so we generally avoid as much as possible. >> > > Fair enough. > >>> Considering >>> these are lightweight exits (on the order of 1-2us), >>> >> APIC EOIs on x86 are MMIO based, so they are generally much heavier than >> that. I measure at least 4-5us just for the MMIO exit on my Woodcrest, >> never mind executing the locking/apic-emulation code. >> > > You won't like to hear me say this, but Woodcrests are pretty old and > clunky as far as VT goes :-) Fair enough. > > On a modern Nehalem, I would be surprised if an MMIO exit handled in the > kernel was muck more than 2us. The hardware is getting very, very > fast. The trends here are very important to consider when we're looking > at architectures that we potentially are going to support for a long time. The exit you do not take will always be infinitely faster. > >>> you need an awfully >>> large amount of interrupts before you get really significant performance >>> impact. You would think NAPI would kick in at this point anyway. >>> >>> >> Whether NAPI can kick in or not is workload dependent, and it also does >> not address coincident events. But on that topic, you can think of >> AlacrityVM's interrupt controller as "NAPI for interrupts", because it >> operates on the same principle. For what its worth, it also operates on >> a "NAPI for hypercalls" concept too. >> > > The concept of always batching hypercalls has certainly b
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 11:33 AM, Andi Kleen wrote: We're not talking about vaporware. vhost-net exists. Is it as fast as the alacrityvm setup then e.g. for network traffic? Last I heard the first could do wirespeed 10Gbit/s on standard hardware. I'm very wary of any such claims. As far as I know, no one has done an exhaustive study of vbus and published the results. This is why it's so important to understand why the results are what they are when we see numbers posted. For instance, check out http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf slide 32. These benchmarks show KVM without vhost-net pretty closely pacing native. With large message sizes, it's awfully close to line rate. Comparatively speaking, consider http://developer.novell.com/wiki/index.php/AlacrityVM/Results vbus here is pretty far off of native and virtio-net is ridiculus. Why are the results so different? Because benchmarking is fickle and networking performance is complicated. No one benchmarking scenario is going to give you a very good picture overall. It's also relatively easy to stack the cards in favor of one approach verses another. The virtio-net setup probably made extensive use of pinning and other tricks to make things faster than a normal user would see them. It ends up creating a perfect combination of batching which is pretty much just cooking the mitigation schemes to do extremely well for one benchmark. This is why it's so important to look at vbus from the perspective of critically asking, what precisely makes it better than virtio. A couple benchmarks on a single piece of hardware does not constitute an existence proof that it's better overall. There are a ton of differences between virtio and vbus because vbus was written in a vacuum wrt virtio. I'm not saying we are totally committed to virtio no matter what, but it should take a whole lot more than a couple netperf runs on a single piece of hardware for a single kind of driver to justify replacing it. Can vhost-net do the same thing? I think the fundamentally question is, what makes vbus better than vhost-net? vhost-net exists and is further along upstream than vbus is at the moment. If that question cannot be answered with technical facts and numbers to back them up, then we're just arguing for the sake of arguing. Regards, Anthony Liguori -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Anthony Liguori (anth...@codemonkey.ws) wrote: > The > virtio-net setup probably made extensive use of pinning and other tricks > to make things faster than a normal user would see them. It ends up > creating a perfect combination of batching which is pretty much just > cooking the mitigation schemes to do extremely well for one benchmark. Just pinning, the rest is stock virtio features like mergeable rx buffers, GRO, GSO (tx mitigation is actually disabled). Certainly doesn't show throughput in terms of cpu cycle cost (scaling) nor latency per-packet (exit and mitigation). thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Tue, 22 Dec 2009, Gregory Haskins wrote: > On 12/22/09 2:39 PM, Davide Libenzi wrote: > > On Tue, 22 Dec 2009, Gregory Haskins wrote: > > > >> On 12/22/09 1:53 PM, Avi Kivity wrote: > >>> I asked why the irqfd/ioeventfd mechanisms are insufficient, and you did > >>> not reply. > >>> > >> > >> BTW: the ioeventfd issue just fell through the cracks, so sorry about > >> that. Note that I have no specific issue with irqfd ever since the > >> lockless IRQ injection code was added. > >> > >> ioeventfd turned out to be suboptimal for me in the fast path for two > >> reasons: > >> > >> 1) the underlying eventfd is called in atomic context. I had posted > >> patches to Davide to address that limitation, but I believe he rejected > >> them on the grounds that they are only relevant to KVM. > > > > I thought we addressed this already, in the few hundreds of email we > > exchanged back then :) > > We addressed the race conditions, but not the atomic callbacks. I can't > remember exactly what you said, but the effect was "no", so I dropped it. ;) > > This was the thread. > > http://www.archivum.info/linux-ker...@vger.kernel.org/2009-06/08548/Re:_[KVM-RFC_PATCH_1_2]_eventfd:_add_an_explicit_srcu_based_notifier_interface Didn't that ended up in schedule_work() being just fine, and no need was there for pre-emptible callbacks? > >> 2) it cannot retain the data field passed in the PIO. I wanted to have > >> one vector that could tell me what value was written, and this cannot be > >> expressed in ioeventfd. > > > > Like might have hinted in his reply, couldn't you add data support to the > > ioeventfd bits in KVM, instead of leaking them into mainline eventfd? > > > > Perhaps, or even easier I could extend xinterface. Which is what I did ;) > > The problem with the first proposal is that you would no longer actually > have an eventfd based mechanism...so any code using ioeventfd (like > Michael Tsirkin's for instance) would break. At that point, the KVM eventfd can take care of thing so that Michael bits do not break. - Davide -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Tue, Dec 22, 2009 at 12:36, Gregory Haskins wrote: > On 12/22/09 2:57 AM, Ingo Molnar wrote: >> * Gregory Haskins wrote: >>> Actually, these patches have nothing to do with the KVM folks. [...] >> >> That claim is curious to me - the AlacrityVM host > > It's quite simple, really. These drivers support accessing vbus, and > vbus is hypervisor agnostic. In fact, vbus isn't necessarily even > hypervisor related. It may be used anywhere where a Linux kernel is the > "io backend", which includes hypervisors like AlacrityVM, but also > userspace apps, and interconnected physical systems as well. > > The vbus-core on the backend, and the drivers on the frontend operate > completely independent of the underlying hypervisor. A glue piece > called a "connector" ties them together, and any "hypervisor" specific > details are encapsulated in the connector module. In this case, the > connector surfaces to the guest side as a pci-bridge, so even that is > not hypervisor specific per se. It will work with any pci-bridge that > exposes a compatible ABI, which conceivably could be actual hardware. This is actually something that is of particular interest to me. I have a few prototype boards right now with programmable PCI-E host/device links on them; one of my long-term plans is to finagle vbus into providing multiple "virtual" devices across that single PCI-E interface. Specifically, I want to be able to provide virtual NIC(s), serial ports and serial consoles, virtual block storage, and possibly other kinds of interfaces. My big problem with existing virtio right now (although I would be happy to be proven wrong) is that it seems to need some sort of out-of-band communication channel for setting up devices, not to mention it seems to need one PCI device per virtual device. So I would love to be able to port something like vbus to my nify PCI hardware and write some backend drivers... then my PCI-E connected systems would dynamically provide a list of highly-efficient "virtual" devices to each other, with only one 4-lane PCI-E bus. Cheers, Kyle Moffett -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Anthony Liguori wrote: > On 12/22/2009 10:01 AM, Bartlomiej Zolnierkiewicz wrote: > >>new e1000 driver is more superior in architecture and do the required > >>work to make the new e1000 driver a full replacement for the old one. > >Right, like everyone actually does things this way.. > > > >I wonder why do we have OSS, old Firewire and IDE stacks still around then? > > And it's always a source of pain, isn't it. Even putting aside the fact that such overlap sucks and is a pain to users (and that 98% of driver and subsystem version transitions are done completely seemlessly to users - the examples that were cited were the odd ones out of 150K commits in the past 4 years, 149K+ of which are seemless), the comparison does not even apply really. e1000, OSS, old Firewire and IDE are hardware stacks, where hardware is a not fully understood externality, with its inevitable set of compatibility voes. There's often situations where one piece of hardware still works better with the old driver, for some odd (or not so odd) reason. Also, note that the 'new' hw drivers are generally intended and are maintained as clear replacements for the old ones, and do so with minimal ABI changes - or preferably with no ABI changes at all. Most driver developers just switch from old to new and the old bits are left around and are phased out. We phased out old OSS recently. That is a very different situation from the AlacrityVM patches, which: - Are a pure software concept and any compatibility mismatch is self-inflicted. The patches are in fact breaking the ABI to KVM intentionally (for better or worse). - Gregory claims that the AlacricityVM patches are not a replacement for KVM. I.e. there's no intention to phase out any 'old stuff' and it splits the pool of driver developers. i.e. it has all the makings of a stupid, avoidable, permanent fork. The thing is, if AlacricityVM is better, and KVM developers are not willing to fix their stuff, replace KVM with it. It's a bit as if someone found a performance problem with sys_open() and came up with sys_open_v2() and claimed that he wants to work with the VFS developers while not really doing so but advances sys_open_v2() all the time. Do we allow sys_open_v2() upstream, in the name of openness and diversity, letting some apps use that syscall while other apps still use sys_open()? Or do we say "enough is enough of this stupidity, come up with some strong reasons to replace sys_open, and if so, replace the thing and be done with the pain!". Overlap and forking can still be done in special circumstances, when a project splits and a hostile fork is inevitable due to prolongued and irreconcilable differences between the parties and if there's no strong technical advantage on either side. I havent seen evidence of this yet though: Gregory claims that he wants to 'work with the community' and the KVM guys seem to agree violently that performance can be improved - and are doing so (and are asking Gregory to take part in that effort). The main difference is that Gregory claims that improved performance is not possible within the existing KVM framework, while the KVM developers disagree. The good news is that this is a hard, testable fact. I think we should try _much_ harder before giving up and forking the ABI of a healthy project and intentionally inflicting pain on our users. And, at minimum, such kinds of things _have to_ be pointed out in pull requests, because it's like utterly important. In fact i couldnt list any more important thing to point out in a pull request. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> i.e. it has all the makings of a stupid, avoidable, permanent fork. The thing Nearly. There was no equivalent of a kernel based virtual driver host before. > - Are a pure software concept and any compatibility mismatch is > > self-inflicted. The patches are in fact breaking the ABI to KVM > In practice, especially considering older kernel releases, VMs behave like hardware, with all its quirks, compatibility requirements, sometimes not fully understood, etc. > is, if AlacricityVM is better, and KVM developers are not willing to fix > their > stuff, replace KVM with it. In the end the driver model is only a very small part of KVM though. > > It's a bit as if someone found a performance problem with sys_open() and came > up with sys_open_v2() and claimed that he wants to work with the VFS > developers while not really doing so but advances sys_open_v2() all the time. AFAIK Gregory tried for several months to work with the KVM maintainers, but failed at their NIH filter. > > Do we allow sys_open_v2() upstream, in the name of openness and diversity, > letting some apps use that syscall while other apps still use sys_open()? Or > do we say "enough is enough of this stupidity, come up with some strong > reasons to replace sys_open, and if so, replace the thing and be done with > the > pain!". I thought the published benchmark numbers were strong reasons. I certainly haven't seen similarly convincing numbers for vhost-net. > The main difference is that Gregory claims that improved performance is not > possible within the existing KVM framework, while the KVM developers > disagree. > The good news is that this is a hard, testable fact. Yes clearly the onus at this point is on the vhost-net developers/ "pci is all that is ever needed for PV" proponents to show similar numbers with their current code. If they can show the same performance there's really no need for the alacrityvm model (or at least I haven't seen a convincing reason other than performance so far to have a separate model) I heard claims earlier from one side to the other that some benchmarks were not fair. Apart from such accusations not being very constructive (I don't think anyone is trying to intentionally mislead others here), but even if that's the case surely the other side can do similar benchmarks and demonstrate they are as fast. -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 12:13 PM, Andi Kleen wrote: i.e. it has all the makings of a stupid, avoidable, permanent fork. The thing Nearly. There was no equivalent of a kernel based virtual driver host before. These are guest drivers. We have virtio drivers, and Xen drivers (which are Xen-specific). - Are a pure software concept and any compatibility mismatch is self-inflicted. The patches are in fact breaking the ABI to KVM In practice, especially considering older kernel releases, VMs behave like hardware, with all its quirks, compatibility requirements, sometimes not fully understood, etc. There was no attempt by Gregory to improve virtio-net. It's a bit as if someone found a performance problem with sys_open() and came up with sys_open_v2() and claimed that he wants to work with the VFS developers while not really doing so but advances sys_open_v2() all the time. AFAIK Gregory tried for several months to work with the KVM maintainers, but failed at their NIH filter. It was the backwards compatibility, live migration, unneeded complexity, and scalability filters from where I sit. vbus fails on all four. The main difference is that Gregory claims that improved performance is not possible within the existing KVM framework, while the KVM developers disagree. The good news is that this is a hard, testable fact. Yes clearly the onus at this point is on the vhost-net developers/ "pci is all that is ever needed for PV" proponents to show similar numbers with their current code. If they can show the same performance there's really no need for the alacrityvm model (or at least I haven't seen a convincing reason other than performance so far to have a separate model) Anthony posted this: http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf See slide 32. This is without vhost-net. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf > > See slide 32. This is without vhost-net. Thanks. Do you also have latency numbers? It seems like there's definitely still potential for improvement with messages <4K. But for the large messages they indeed look rather good. It's unclear what message size the Alacrity numbers used, but I presume it was rather large. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 02:14 PM, Andi Kleen wrote: http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf See slide 32. This is without vhost-net. Thanks. Do you also have latency numbers? No. Copying Chris. This was with the tx mitigation timer disabled, so you won't see the usual atrocious userspace virtio latencies, but it won't be as good as a host kernel implementation since we take a heavyweight exit and qemu is pretty unoptimized. It seems like there's definitely still potential for improvement with messages<4K. But for the large messages they indeed look rather good. There's still a lot of optimization to be done, but I hope this proves there is nothing inherently slow about virtio. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wednesday 23 December 2009 07:51:29 am Ingo Molnar wrote: > > * Anthony Liguori wrote: > > > On 12/22/2009 10:01 AM, Bartlomiej Zolnierkiewicz wrote: > > >>new e1000 driver is more superior in architecture and do the required > > >>work to make the new e1000 driver a full replacement for the old one. > > >Right, like everyone actually does things this way.. > > > > > >I wonder why do we have OSS, old Firewire and IDE stacks still around then? > > > > And it's always a source of pain, isn't it. > > Even putting aside the fact that such overlap sucks and is a pain to users > (and that 98% of driver and subsystem version transitions are done completely > seemlessly to users - the examples that were cited were the odd ones out of > 150K commits in the past 4 years, 149K+ of which are seemless), the > comparison > does not even apply really. Total commit number has nothing to do with the issue raised since the problem is in the total source code complexity and the need to maintain separate code bases. [ BTW I find your habit of bringing the completely unrelated numbers into the discussion quite annoying. Do you really think that throwing in some random numbers automatically increases the credibility of your opinion? ] > e1000, OSS, old Firewire and IDE are hardware stacks, where hardware is a not > fully understood externality, with its inevitable set of compatibility voes. > There's often situations where one piece of hardware still works better with > the old driver, for some odd (or not so odd) reason. > > Also, note that the 'new' hw drivers are generally intended and are > maintained > as clear replacements for the old ones, and do so with minimal ABI changes - > or preferably with no ABI changes at all. Most driver developers just switch > from old to new and the old bits are left around and are phased out. We > phased > out old OSS recently. 'We' as Fedora? old OSS stuff is still there (sound/oss/ which is almost 45KLOC and would be much more if not past efforts from Adrian Bunk to shrink it down) Besides 'phase out' that you are talking about comes down to just waiting for old hardware/user base to die and it takes years to accomplish.. I can understand how this is not an issue from i.e. Red Hat's POV when you have one 'set in the stone' set of drivers in RHEL and the other 'constant development flux' one in Fedora (which because of this fact cannot be no longer consider as real distribution for real users BTW) but for everybody else this is simply untrue. > That is a very different situation from the AlacrityVM patches, which: > > - Are a pure software concept and any compatibility mismatch is >self-inflicted. The patches are in fact breaking the ABI to KVM >intentionally (for better or worse). Care to explain the 'breakage' and why KVM is more special in this regard than other parts of the kernel (where we don't keep any such requirements)? Truth to be told KVM is just another driver/subsystem and Gregory's changes are only 4KLOC of clean and easily maintainable code.. > - Gregory claims that the AlacricityVM patches are not a replacement for KVM. >I.e. there's no intention to phase out any 'old stuff' and it splits the >pool of driver developers. Talk double standards. It was you & co. that officially legitimized this style of doing things and now you are complaining about it? > i.e. it has all the makings of a stupid, avoidable, permanent fork. The thing > is, if AlacricityVM is better, and KVM developers are not willing to fix > their > stuff, replace KVM with it. > > It's a bit as if someone found a performance problem with sys_open() and came > up with sys_open_v2() and claimed that he wants to work with the VFS > developers while not really doing so but advances sys_open_v2() all the time. > > Do we allow sys_open_v2() upstream, in the name of openness and diversity, > letting some apps use that syscall while other apps still use sys_open()? Or > do we say "enough is enough of this stupidity, come up with some strong > reasons to replace sys_open, and if so, replace the thing and be done with > the > pain!". I certainly missed the time when KVM became officially part of core ABI.. > Overlap and forking can still be done in special circumstances, when a > project > splits and a hostile fork is inevitable due to prolongued and irreconcilable > differences between the parties and if there's no strong technical advantage > on either side. I havent seen evidence of this yet though: Gregory claims > that > he wants to 'work with the community' and the KVM guys seem to agree > violently > that performance can be improved - and are doing so (and are asking Gregory > to > take part in that effort). How it is different from any past forks? The odium of proving that the existing framework is sufficient was always on original authors or current maintainers. KVM guys were offered assistance from Gregory and had few mon
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 03:07 PM, Bartlomiej Zolnierkiewicz wrote: That is a very different situation from the AlacrityVM patches, which: - Are a pure software concept and any compatibility mismatch is self-inflicted. The patches are in fact breaking the ABI to KVM intentionally (for better or worse). Care to explain the 'breakage' and why KVM is more special in this regard than other parts of the kernel (where we don't keep any such requirements)? The device model is exposed to the guest. If you change it, the guest breaks. So we have two options: - phase out virtio, users don't see new improvements, ask them to change to vbus/venet - maintain the two in parallel Neither appeals to me. Truth to be told KVM is just another driver/subsystem and Gregory's changes are only 4KLOC of clean and easily maintainable code.. This 4K is only the beginning. There are five more virtio drivers, plus features in virtio-net not ported to venet, plus the host support, plus qemu support, plus Windows drivers, plus adapters for non-pci (lguest and s390), plus live migration support. vbus itself still has scaling issues. Virtio was under development for years. Sure you can focus on one dimension only (performance) and get good results but real life is more complicated. I certainly missed the time when KVM became officially part of core ABI.. It's more akin to the hardware interface. We don't change the hardware underneath the guest. Overlap and forking can still be done in special circumstances, when a project splits and a hostile fork is inevitable due to prolongued and irreconcilable differences between the parties and if there's no strong technical advantage on either side. I havent seen evidence of this yet though: Gregory claims that he wants to 'work with the community' and the KVM guys seem to agree violently that performance can be improved - and are doing so (and are asking Gregory to take part in that effort). How it is different from any past forks? The odium of proving that the existing framework is sufficient was always on original authors or current maintainers. KVM guys were offered assistance from Gregory and had few months to prove that they can get the same kind of performance using existing architecture and they DID NOT do it. Look at the results from Chris Wright's presentation. Hopefully in a few days some results from vhost-net. Then please try harder. Gregory posted his initial patches in August, it is December now and we only see artificial road-blocks instead of code from KVM folks. What artificial road blocks? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wednesday 23 December 2009 02:31:11 pm Avi Kivity wrote: > On 12/23/2009 03:07 PM, Bartlomiej Zolnierkiewicz wrote: > > > >> That is a very different situation from the AlacrityVM patches, which: > >> > >> - Are a pure software concept and any compatibility mismatch is > >> self-inflicted. The patches are in fact breaking the ABI to KVM > >> intentionally (for better or worse). > >> > > Care to explain the 'breakage' and why KVM is more special in this regard > > than other parts of the kernel (where we don't keep any such requirements)? > > > > The device model is exposed to the guest. If you change it, the guest > breaks. Huh? Shouldn't non-vbus aware guests continue to work just fine? > > I certainly missed the time when KVM became officially part of core ABI.. > > > > It's more akin to the hardware interface. We don't change the hardware > underneath the guest. As far as my limited understanding of things go vbus is completely opt-in so it is like adding new real hardware to host. Where is the problem? -- Bartlomiej Zolnierkiewicz -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 04:08 PM, Bartlomiej Zolnierkiewicz wrote: The device model is exposed to the guest. If you change it, the guest breaks. Huh? Shouldn't non-vbus aware guests continue to work just fine? Sure. But we aren't merging this code in order not to use it. If we switch development focus to vbus, we have to ask everyone who's riding on virtio to switch. Alternatively we maintain both models. If vbus was the only way to get this kind of performance, I know what I'd choose. But if it isn't, why inflict the change on users? Consider a pxe-booting guest (or virtio-blk vs. a future veblk). Is switching drivers in initrd something you want your users to do? [1] One of the advantages of virtualization is stable hardware. I don't want to let it go without a very good reason. [1] I remember the move from /dev/hda to /dev/sda a few years ago, it isn't a good memory. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 07:07 AM, Bartlomiej Zolnierkiewicz wrote: On Wednesday 23 December 2009 07:51:29 am Ingo Molnar wrote: KVM guys were offered assistance from Gregory and had few months to prove that they can get the same kind of performance using existing architecture and they DID NOT do it. With all due respect, there is a huge misunderstanding that's unpinning this thread which is that vbus is absolutely more performant than virtio-net and that we've failed to demonstrate that we can obtain the same level of performance in virtio-net. This is simply untrue. In fact, within a week or so of Greg's first posting of vbus, I posted a proof of concept patch to the virtio-net backend that got equivalent results. But I did not feel at the time that this was the right solution to the problem and we've been trying to do something much better. By the same token, I don't feel that vbus is the right approach to solving the problem. There are really three factors that affect networking performance in a virtual environment: the number of copies of the data, the number of exits required per-packet transmission, and the cost of each exit. The "poor" packet latency of virtio-net is a result of the fact that we do software timer based TX mitigation. We do this such that we can decrease the number of exits per-packet and increase throughput. We set a timer for 250ms and per-packet latency will be at least that much. We have to use a timer for the userspace backend because the tun/tap device is rather quick to queue a packet which means that we get no feedback that we can use to trigger TX mitigation. vbus works around this by introducing a transmit and receive thread and relies on the time it takes to schedule those threads to do TX mitigation. The version of KVM in RHEL5.4 does the same thing. How effective this is depends on a lot of factors including the overall system load, the default time slice length, etc. This tends to look really good when you're trying to drive line speed but it absolutely sucks when you're looking at the CPU cost of low packet rates. IOW, this is a heuristic that looks really good when doing netperf TCP_RR and TCP_STREAM, but it starts to look really bad when doing things like partial load CPU usage comparisons with other hypervisors. vhost-net takes a different, IMHO superior, approach in that it associates with some type of network device (tun/tap or physical device) and uses the device's transmit interface to determine how to mitigate packets. This means that we can potentially get to the point where instead of relying on short timeouts to do TX mitigation, we can use the underlying physical device's packet processing state which will provide better results in most circumstances. N.B. using a separate thread for transmit mitigation looks really good on benchmarks because when doing a simple ping test, you'll see very short latencies because you're not batching at all. It's somewhat artificial in this regard. With respect to number of copies, vbus up until recently had the same number of copies as virtio-net. Greg has been working on zero-copy transmit, which is great stuff, but Rusty Russell had done the same thing with virtio-net and tun/tap. There are some hidden nasties when using skb destructors to achieve this and I think the feeling was this wasn't going to work. Hopefully, Greg has better luck but suffice to say, we've definitely demonstrated this before with virtio-net. If the issues around skb destruction can be resolved, we can incorporate this into tun/tap (and therefore, use it in virtio) very easily. In terms of the cost per exit, the main advantage vbus had over virtio-net was that virtio-net's userspace backend was in userspace which required a heavy-weight exit which is a few times more expensive than a lightweight exit. We've addressed this with vhost-net which implements the backend in the kernel. Originally, vbus was able to do edge triggered interrupts whereas virtio-pci was using level triggered interrupts. We've since implemented MSI-X support (already merged upstream) and we now can also do edge triggered interrupts with virtio. The only remaining difference is the fact that vbus can mitigate exits due to EOI's in the virtual APIC because it relies on a paravirtual interrupt controller. This is rather controversial for a few reasons. The first is that there is absolutely no way that a paravirtual interrupt controller would work for Windows, older Linux guests, or probably any non-Linux guest. As a design point, this is a big problem for KVM. We've seen the struggle with this sort of thing with Xen. The second is that it's very likely that this problem will go away on it's own either because we'll rely on x2apic (which will eventually work with Windows) or we'll see better hardware support for eoi shadowing (there is already hardware support for tpr shadowing). Mos
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 06:14 AM, Andi Kleen wrote: http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf See slide 32. This is without vhost-net. Thanks. Do you also have latency numbers? They'll be along the lines of the vbus numbers. But I caution people from relying too much on just netperf TCP_RR and TCP_STREAM numbers. There's a lot of heuristics in play in getting this sort of numbers. They really aren't good ways to compare different drivers. A better thing to do is look more deeply at the architectures and consider things like the amount of copying imposed, the cost of processing an exit, and the mechanisms for batching packet transmissions. The real argument that vbus needs to make IMHO is not "look how much better my netperf TCP_STREAM results are" but "we can eliminate N copies from the transmit path and virtio-net cannot" or "we require N exits to handle a submission and virtio-net requires N+M". It's too easy to tweak for benchmarks. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 12:15 AM, Kyle Moffett wrote: This is actually something that is of particular interest to me. I have a few prototype boards right now with programmable PCI-E host/device links on them; one of my long-term plans is to finagle vbus into providing multiple "virtual" devices across that single PCI-E interface. Specifically, I want to be able to provide virtual NIC(s), serial ports and serial consoles, virtual block storage, and possibly other kinds of interfaces. My big problem with existing virtio right now (although I would be happy to be proven wrong) is that it seems to need some sort of out-of-band communication channel for setting up devices, not to mention it seems to need one PCI device per virtual device. We've been thinking about doing a virtio-over-IP mechanism such that you could remote the entire virtio bus to a separate physical machine. virtio-over-IB is probably more interesting since you can make use of RDMA. virtio-over-PCI-e would work just as well. virtio is a layered architecture. Device enumeration/discovery sits at a lower level than the actual device ABIs. The device ABIs are implemented on top of a bulk data transfer API. The reason for this layering is so that we can reuse PCI as an enumeration/discovery mechanism. This tremendenously simplifies porting drivers to other OSes and let's us use PCI hotplug automatically. We get integration into all the fancy userspace hotplug support for free. But both virtio-lguest and virtio-s390 use in-band enumeration and discovery since they do not have support for PCI on either platform. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/22/2009 06:02 PM, Chris Wright wrote: * Anthony Liguori (anth...@codemonkey.ws) wrote: The virtio-net setup probably made extensive use of pinning and other tricks to make things faster than a normal user would see them. It ends up creating a perfect combination of batching which is pretty much just cooking the mitigation schemes to do extremely well for one benchmark. Just pinning, the rest is stock virtio features like mergeable rx buffers, GRO, GSO (tx mitigation is actually disabled). Technically, tx mitigation isn't disabled. The heuristic is changed such that instead of relying on a fixed timer, tx notification is disabled until you can switch to another thread and process packets. The effect is that depending on time slice length and system load, you adaptively enable tx mitigation. It's heavily dependent on the particulars of the system and the overall load. For instance, this mitigation scheme looks great at high throughputs but looks very bad at mid-to-low throughputs compared to timer based mitigation (at least, when comparing CPU cost). Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Anthony Liguori (anth...@codemonkey.ws) wrote: > On 12/22/2009 06:02 PM, Chris Wright wrote: >> * Anthony Liguori (anth...@codemonkey.ws) wrote: >>> The >>> virtio-net setup probably made extensive use of pinning and other tricks >>> to make things faster than a normal user would see them. It ends up >>> creating a perfect combination of batching which is pretty much just >>> cooking the mitigation schemes to do extremely well for one benchmark. >> >> Just pinning, the rest is stock virtio features like mergeable rx buffers, >> GRO, GSO (tx mitigation is actually disabled). > > Technically, tx mitigation isn't disabled. The heuristic is changed > such that instead of relying on a fixed timer, tx notification is > disabled until you can switch to another thread and process packets. > > The effect is that depending on time slice length and system load, you > adaptively enable tx mitigation. It's heavily dependent on the > particulars of the system and the overall load. > > For instance, this mitigation scheme looks great at high throughputs but > looks very bad at mid-to-low throughputs compared to timer based > mitigation (at least, when comparing CPU cost). Yep, you're right. thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 1:51 AM, Ingo Molnar wrote: > > * Anthony Liguori wrote: > >> On 12/22/2009 10:01 AM, Bartlomiej Zolnierkiewicz wrote: new e1000 driver is more superior in architecture and do the required work to make the new e1000 driver a full replacement for the old one. >>> Right, like everyone actually does things this way.. >>> >>> I wonder why do we have OSS, old Firewire and IDE stacks still around then? >> >> And it's always a source of pain, isn't it. > > Even putting aside the fact that such overlap sucks and is a pain to users > (and that 98% of driver and subsystem version transitions are done completely > seemlessly to users - the examples that were cited were the odd ones out of > 150K commits in the past 4 years, 149K+ of which are seemless), the > comparison > does not even apply really. > > e1000, OSS, old Firewire and IDE are hardware stacks, where hardware is a not > fully understood externality, with its inevitable set of compatibility voes. > There's often situations where one piece of hardware still works better with > the old driver, for some odd (or not so odd) reason. > > Also, note that the 'new' hw drivers are generally intended and are > maintained > as clear replacements for the old ones, and do so with minimal ABI changes - > or preferably with no ABI changes at all. Most driver developers just switch > from old to new and the old bits are left around and are phased out. We > phased > out old OSS recently. > > That is a very different situation from the AlacrityVM patches, which: > > - Are a pure software concept By design. In fact, I would describe it as "software to software optimized" as opposed to trying to shoehorn into something that was designed as a software-to-hardware interface (and therefore has assumptions about the constraints in that environment that are not applicable in software-only). > and any compatibility mismatch is self-inflicted. .. because the old model is not great for the intended use cases and has issues. I've already covered the reasons why adnauseam. > The patches are in fact breaking the ABI to KVM intentionally (for better or > worse). No, at the very worst they are _augmenting_ the ABI, as evident from the fact that AlacrityVM is a superset of the entire KVM ABI. > > - Gregory claims that the AlacricityVM patches are not a replacement for KVM. >I.e. there's no intention to phase out any 'old stuff' There's no reason to phase anything out, except perhaps the virtio-pci transport. This is one more transport, plugging into virtio underneath (just like virtio-pci, virtio-lguest, and virtio-s390). I am not even suggesting that the old transport has to go away, per se. It is the KVM maintainers who insist on it being all or nothing. For me, I do not see the big deal in having one more "model" option in the qemu cmd-line, but that is just my opinion. If the maintainers were really so adamant that choice is pure evil, I am not sure why we don't see patches for removing everything but one model type in each IO category. But I digress. > and it splits the pool of driver developers. ..it these dumb threads that are splitting driver developers with ignorant statements, irrelevant numbers, and dubious "facts". I actually tried many many times to ask others to join the effort, and instead _they_ forked off and made vhost-net with a "sorry, not interested in working with you"(and based the design largely on the ideas proposed in my framework, I might add). Thats fine, and it's their prerogative. I can easily maintain my project out of tree if upstream is not interested. But do not turn around and try to blame me for the current state of affairs. > > i.e. it has all the makings of a stupid, avoidable, permanent fork. The thing > is, if AlacricityVM is better, and KVM developers are not willing to fix > their > stuff, replace KVM with it. > > It's a bit as if someone found a performance problem with sys_open() and came > up with sys_open_v2() and claimed that he wants to work with the VFS > developers while not really doing so but advances sys_open_v2() all the time. No, its more like if I suggested sys_open_vbus() to augment sys_open_pci(), sys_open_lguest(), sys_open_s390() because our fictitious glibc natively supported modular open() implementations. And I tried to work for a very long time convincing the VFS developers that this new transport was a good idea to add because it was optimized for the problem space, made it easy for API callers to reuse the important elements of the design (e.g. built in "tx-mitigation" instead of waiting for someone to write it for each device), had new features like the ability to prioritize signals, create/route signal paths arbitrarily, implement raw shared memory for idempotent updates, and didn't require the massive and useless PCI/APIC emulation logic to function like sys_open_pci() (e.g. easier to port around). Ultimately, the "VFS developers" said "
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Avi Kivity (a...@redhat.com) wrote: > On 12/23/2009 02:14 PM, Andi Kleen wrote: > >>http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf > >> > >>See slide 32. This is without vhost-net. > >Thanks. Do you also have latency numbers? > > No. Copying Chris. This was with the tx mitigation timer disabled, > so you won't see the usual atrocious userspace virtio latencies, but > it won't be as good as a host kernel implementation since we take a > heavyweight exit and qemu is pretty unoptimized. Those numbers don't show cpu cycles per packet nor do they show latencies. You won't see the timer based latency, because the tx mitigation scheme is not timer based for those numbers. Below are some numbers comparing bare metal, an assigned device, and virtio (not vhost-net, so we are still doing a heavy-weight exit to qemu and syscalls to deliver to tap device). > >It seems like there's definitely still potential for improvement > >with messages<4K. But for the large messages they indeed > >look rather good. You are misreading the graph. At <4K it is tracking bare metal (the green and yellow lines are bare metal, the red and blue bars are virtio). At >4k we start to drop off (esp. on RX). This (slide 9) shows AMQP latencies for bare metal, an assigned device, and virtio. http://www.redhat.com/f/pdf/summit/bche_320_red_hat_enterprise_mrg.pdf Similarly, here's some much rawer latency numbers from netpipe, all done in usecs. bareassigned metal PCI NIC virtio (usecs) (usecs) (usecs) - - - 1 bytes 22.20 36.16 53.19 2 bytes 22.21 35.98 53.23 3 bytes 22.22 36.18 53.29 4 bytes 22.25 33.77 53.43 6 bytes 22.33 36.33 53.48 8 bytes 22.32 36.24 53.27 12 bytes 22.25 35.97 53.33 13 bytes 22.40 35.94 53.54 16 bytes 22.36 35.98 53.60 19 bytes 22.40 35.95 53.51 21 bytes 22.42 35.94 53.76 24 bytes 22.32 36.18 53.45 27 bytes 22.34 36.08 53.48 29 bytes 22.36 36.02 53.42 32 bytes 22.46 36.15 53.23 35 bytes 22.36 36.23 53.13 45 bytes 26.32 36.17 53.29 48 bytes 26.24 35.94 53.50 51 bytes 26.44 36.01 53.66 61 bytes 26.43 33.66 53.28 64 bytes 26.66 36.32 53.17 67 bytes 26.35 36.21 53.53 93 bytes 26.59 36.49 45.75 96 bytes 26.48 36.28 45.72 99 bytes 26.51 36.47 45.72 125 bytes 26.74 36.48 45.99 128 bytes 26.44 36.52 45.69 131 bytes 26.52 35.71 45.80 189 bytes 26.77 36.99 46.78 192 bytes 26.96 37.45 47.00 195 bytes 26.96 37.45 47.10 253 bytes 27.01 38.03 47.36 256 bytes 27.09 37.85 47.23 259 bytes 26.98 37.82 47.28 381 bytes 26.61 38.38 47.84 384 bytes 26.72 38.54 48.01 387 bytes 26.76 38.65 47.80 509 bytes 25.13 39.19 48.30 512 bytes 25.13 36.69 56.05 515 bytes 25.15 37.42 55.70 765 bytes 25.29 40.31 57.26 768 bytes 25.25 39.76 57.32 771 bytes 25.26 40.33 57.06 1021 bytes 49.27 57.00 63.73 1024 bytes 49.33 57.09 63.70 1027 bytes 49.07 57.25 63.70 1533 bytes 50.11 58.98 70.57 1536 bytes 50.09 59.30 70.22 1539 bytes 50.18 59.27 70.35 2045 bytes 50.44 59.42 74.31 2048 bytes 50.33 59.29 75.31 2051 bytes 50.32 59.14 74.02 3069 bytes 62.71 64.20 96.87 3072 bytes 62.78 64.94 96.84 3075 bytes 62.83 65.13 96.62 4093 bytes 62.56 64.78 99.63 4096 bytes 62.46 65.04 99.54 4099 bytes 62.47 65.87 99.65 6141 bytes 63.35 65.39 104.03 6144 bytes 63.59 66.16 104.66 6147 bytes 63.74 66.04 104.61 8189 bytes 63.65 66.52 107.75 8192 bytes 63.64 66.71 108.17 8195 bytes 63.66 67.08 109.11 12285 bytes 63.26 84.58 114.13 12288 bytes 63.28 85.38 114.55 12291 bytes 63.22 83.71 114.40 16381 bytes 62.87 98.19 120.48 16384 bytes 63.12 97.96 122.19 16387 bytes 63.48 98.48 121.68 24573 bytes 93.26 108.93 152.67 24576 bytes 94.40 109.42 152.14 24579 bytes 93.37 108.86 153.51 32765 bytes 102.84 115.46 169.04 32768 bytes 100.01 114.62 166.19 32771 bytes 102.61 115.97 167.96 49149 bytes 125.46 144.78 209.99 49152 bytes 123.76 139.70 187.17 49155 bytes 125.13 137.97 185.44 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> And its moot, anyway, as I have already retracted my one outstanding > pull request based on Linus' observation. So at this time, I am not > advocating _anything_ for upstream inclusion. And I am contemplating > _never_ doing so again. It's not worth _this_. That certainly sounds like the wrong reaction. Out of tree drivers are typically a pain to use. And upstream submission is not always like this! -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 12:10 PM, Andi Kleen wrote: >> And its moot, anyway, as I have already retracted my one outstanding >> pull request based on Linus' observation. So at this time, I am not >> advocating _anything_ for upstream inclusion. And I am contemplating >> _never_ doing so again. It's not worth _this_. > > That certainly sounds like the wrong reaction. Out of tree drivers > are typically a pain to use. Well, to Linus' point, it shouldn't go in until a critical mass of users have expressed desire to see it in tree, which seems reasonable to me. For the admittedly small group that are using it today, modern tools like the opensuse-build-service ease the deployment as a KMP, so that can suffice for now. Its actually what most of the alacrityvm community uses today anyway (as opposed to using a merged tree in the guest) > > And upstream submission is not always like this! I would think the process would come to a grinding halt if it were ;) Thanks Andi, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> > >It seems like there's definitely still potential for improvement > > >with messages<4K. But for the large messages they indeed > > >look rather good. > > You are misreading the graph. At <4K it is tracking bare metal (the > green and yellow lines are bare metal, the red and blue bars are virtio). > At >4k we start to drop off (esp. on RX). I see. True. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, 23 Dec 2009, Gregory Haskins wrote: > > > > And upstream submission is not always like this! > > I would think the process would come to a grinding halt if it were ;) Well, in all honesty, if it had been non-virtualized drivers I would just have pulled. The pull request all looked sane, the diffstat looked clean and non-intrusive, and I had no problems with any of that. But the virtualization people always argue about the fifty-eleven different ways of doing things, and unlike real drivers - where the actual hardware places constraints on what the heck is going on - virtualization people seem to revel in making new interfaces weekly, and tend to be only incidentally limited by hardware (ie hardware interfaces may limit some _details_, but seldom any higher-level arguments). So when I see another virtualization interface, I want the virtualization people to just argue it out amongst themselves. Thanks to the virtue of me personally not caring one whit about virtualization, I can stand back and just watch the fireworks. Which is not to say that I enjoy it (I like the occasional flame-fest, but in order to like them I need to _care_ enough to get fired up about them!). So I just don't want the in-fighting to take place in my tree, so I'd rather see the fighting die out _before_ I actually pull. You people are all crazy. Linus -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 12:17:48PM -0500, Gregory Haskins wrote: > On 12/23/09 12:10 PM, Andi Kleen wrote: > >> And its moot, anyway, as I have already retracted my one outstanding > >> pull request based on Linus' observation. So at this time, I am not > >> advocating _anything_ for upstream inclusion. And I am contemplating > >> _never_ doing so again. It's not worth _this_. > > > > That certainly sounds like the wrong reaction. Out of tree drivers > > are typically a pain to use. > > Well, to Linus' point, it shouldn't go in until a critical mass of users > have expressed desire to see it in tree, which seems reasonable to me. > For the admittedly small group that are using it today, modern tools > like the opensuse-build-service ease the deployment as a KMP, so that > can suffice for now. Its actually what most of the alacrityvm community > uses today anyway (as opposed to using a merged tree in the guest) It would be probably also good to have some more exhaustive data showing any performance improvements. Your numbers are very hard to compare to Chris' numbers and not as comprehensive (e.g. no latencies) -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 1:15 AM, Kyle Moffett wrote: > On Tue, Dec 22, 2009 at 12:36, Gregory Haskins > wrote: >> On 12/22/09 2:57 AM, Ingo Molnar wrote: >>> * Gregory Haskins wrote: Actually, these patches have nothing to do with the KVM folks. [...] >>> >>> That claim is curious to me - the AlacrityVM host >> >> It's quite simple, really. These drivers support accessing vbus, and >> vbus is hypervisor agnostic. In fact, vbus isn't necessarily even >> hypervisor related. It may be used anywhere where a Linux kernel is the >> "io backend", which includes hypervisors like AlacrityVM, but also >> userspace apps, and interconnected physical systems as well. >> >> The vbus-core on the backend, and the drivers on the frontend operate >> completely independent of the underlying hypervisor. A glue piece >> called a "connector" ties them together, and any "hypervisor" specific >> details are encapsulated in the connector module. In this case, the >> connector surfaces to the guest side as a pci-bridge, so even that is >> not hypervisor specific per se. It will work with any pci-bridge that >> exposes a compatible ABI, which conceivably could be actual hardware. > > This is actually something that is of particular interest to me. I > have a few prototype boards right now with programmable PCI-E > host/device links on them; one of my long-term plans is to finagle > vbus into providing multiple "virtual" devices across that single > PCI-E interface. > > Specifically, I want to be able to provide virtual NIC(s), serial > ports and serial consoles, virtual block storage, and possibly other > kinds of interfaces. My big problem with existing virtio right now > (although I would be happy to be proven wrong) is that it seems to > need some sort of out-of-band communication channel for setting up > devices, not to mention it seems to need one PCI device per virtual > device. > > So I would love to be able to port something like vbus to my nify PCI > hardware and write some backend drivers... then my PCI-E connected > systems would dynamically provide a list of highly-efficient "virtual" > devices to each other, with only one 4-lane PCI-E bus. Hi Kyle, We indeed have others that are doing something similar. I have CC'd Ira who may be able to provide you more details. I would also point you at the canonical example for what you would need to write to tie your systems together. Its the "null connector", which you can find here: http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=blob;f=kernel/vbus/connectors/null.c;h=b6d16cb68b7e49e07528278bc9f5b73e1dac0c2f;hb=HEAD Do not hesitate to ask any questions, though you may want to take the conversation to the alacrityvm-devel list as to not annoy the current CC list any further than I already have ;) Kind Regards, -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Anthony Liguori (anth...@codemonkey.ws) wrote: > The "poor" packet latency of virtio-net is a result of the fact that we > do software timer based TX mitigation. We do this such that we can > decrease the number of exits per-packet and increase throughput. We set > a timer for 250ms and per-packet latency will be at least that much. Actually that's 150us ;-) It's the AlacrityVM numbers that show 250us (note micro, not milli) for latency. That makes sense, shave off 150us for the timer and you're left w/ 100us, which is not substantially slower than what we see (for that bare metal latency we see ~60us) when we switched tx mitigation schemes from timer based to thread scheduling. Quite similar to the 56.8us that vbus/venet shows. thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Andi Kleen wrote: > > - Are a pure software concept and any compatibility mismatch is > > > > self-inflicted. The patches are in fact breaking the ABI to KVM > > > > In practice, especially considering older kernel releases, VMs behave like > hardware, with all its quirks, compatibility requirements, sometimes not > fully understood, etc. I stopped reading your reply here. That's not actually fully true of KVM, at all. Virtualization isnt voodoo magic with some hidden souce in some magic hardware component that no-one can understand fully. This isnt some mystic hardware vendor coming up with some code and going away in the next quarter, with barely anything documented and thousands of users left with hardware components which we need to support under Linux somehow. This is Linux virtualization, where _both_ the host and the guest source code is fully known, and bugs (if any) can be found with a high degree of determinism. This is Linux where the players dont just vanish overnight, and are expected to do a proper job. Yes, there's (obviously) compatibility requirements and artifacts and past mistakes (as with any software interface), but you need to admit it to yourself that your "virtualization is sloppy just like hardware" claim is just a cheap excuse to not do a proper job of interface engineering. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
Ingo Molnar writes: > Yes, there's (obviously) compatibility requirements and artifacts and past > mistakes (as with any software interface), but you need to admit it to Yes that's exactly what I meant. > yourself that your "virtualization is sloppy just like hardware" claim is > just In my experience hardware is a lot less sloppy than software. Imagine your latest CPU had as many regressions as 2.6.32 @) I wish software and even VMs were as good. > a cheap excuse to not do a proper job of interface engineering. Past mistakes cannot be easily fixed. And undoubtedly even the new shiny interfaces will have bugs and problems. Also the behaviour is often not completely understood. Maybe it can be easier debugged with fully available source, but even then it's hard to fix the old software (or rather even if you can fix it deploy the fixes). In that regard it's a lot like hardware. I agree with you that this makes it important to design good interfaces, but again realistically mistakes will be made and they cannot be all fixed retroactively. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 12:34:44PM -0500, Gregory Haskins wrote: > On 12/23/09 1:15 AM, Kyle Moffett wrote: > > On Tue, Dec 22, 2009 at 12:36, Gregory Haskins > > wrote: > >> On 12/22/09 2:57 AM, Ingo Molnar wrote: > >>> * Gregory Haskins wrote: > Actually, these patches have nothing to do with the KVM folks. [...] > >>> > >>> That claim is curious to me - the AlacrityVM host > >> > >> It's quite simple, really. These drivers support accessing vbus, and > >> vbus is hypervisor agnostic. In fact, vbus isn't necessarily even > >> hypervisor related. It may be used anywhere where a Linux kernel is the > >> "io backend", which includes hypervisors like AlacrityVM, but also > >> userspace apps, and interconnected physical systems as well. > >> > >> The vbus-core on the backend, and the drivers on the frontend operate > >> completely independent of the underlying hypervisor. A glue piece > >> called a "connector" ties them together, and any "hypervisor" specific > >> details are encapsulated in the connector module. In this case, the > >> connector surfaces to the guest side as a pci-bridge, so even that is > >> not hypervisor specific per se. It will work with any pci-bridge that > >> exposes a compatible ABI, which conceivably could be actual hardware. > > > > This is actually something that is of particular interest to me. I > > have a few prototype boards right now with programmable PCI-E > > host/device links on them; one of my long-term plans is to finagle > > vbus into providing multiple "virtual" devices across that single > > PCI-E interface. > > > > Specifically, I want to be able to provide virtual NIC(s), serial > > ports and serial consoles, virtual block storage, and possibly other > > kinds of interfaces. My big problem with existing virtio right now > > (although I would be happy to be proven wrong) is that it seems to > > need some sort of out-of-band communication channel for setting up > > devices, not to mention it seems to need one PCI device per virtual > > device. > > Greg, thanks for CC'ing me. Hello Kyle, I've got a similar situation here. I've got many PCI agents (devices) plugged into a PCI backplane. I want to use the network to communicate from the agents to the PCI master (host system). At the moment, I'm using a custom driver, heavily based on the PCINet driver posted on the linux-netdev mailing list. David Miller rejected this approach, and suggested I use virtio instead. My first approach with virtio was to create a "crossed-wires" driver, which connected two virtio-net drivers together. While this worked, it doesn't support feature negotiation properly, and so it was scrapped. You can find this posted on linux-netdev with the title "virtio-over-PCI". I started writing a "virtio-phys" layer which creates the appropriate distinction between frontend (guest driver) and backend (kvm, qemu, etc.). This effort has been put on hold for lack of time, and because there is no example code which shows how to create an interface from virtio rings to TUN/TAP. The vhost-net driver is supposed to fill this role, but I haven't seen any test code for that either. The developers haven't been especially helpful answering questions like: how would I use vhost-net with a DMA engine. (You'll quickly find that you must use DMA to transfer data across PCI. AFAIK, CPU's cannot do burst accesses to the PCI bus. I get a 10+ times speedup using DMA.) The virtio-phys work is mostly lacking a backend for virtio-net. It is still incomplete, but at least devices can be registered, etc. It is available at: http://www.mmarray.org/~iws/virtio-phys/ Another thing you'll notice about virtio-net (and vbus' venet) is that they DO NOT specify endianness. This means that they cannot be used with a big-endian guest and a little-endian host, or vice versa. This means they will not work in certain QEMU setups today. Another problem with virtio is that you'll need to invent your own bus model. QEMU/KVM has their bus model, lguest uses a different one, and s390 uses yet another, IIRC. At least vbus provides a standardized bus model. All in all, I've written a lot of virtio code, and it has pretty much all been shot down. It isn't very encouraging. > > So I would love to be able to port something like vbus to my nify PCI > > hardware and write some backend drivers... then my PCI-E connected > > systems would dynamically provide a list of highly-efficient "virtual" > > devices to each other, with only one 4-lane PCI-E bus. I've written some IOQ test code, all of which is posted on the alacrityvm-devel mailing list. If we can figure out how to make IOQ use the proper ioread32()/iowrite32() accessors for accessing ioremap()ed PCI BARs, then I can pretty easily write the rest of a "vbus-phys" connector. > > Hi Kyle, > > We indeed have others that are doing something similar. I have CC'd Ira > who may be able to provide you more details. I would also point you at > the canonical example for wha
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
"Ira W. Snyder" writes: > (You'll quickly find that you must use DMA to transfer data across PCI. > AFAIK, CPU's cannot do burst accesses to the PCI bus. I get a 10+ times AFAIK that's what write-combining on x86 does. DMA has other advantages of course. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 09:27 PM, Andi Kleen wrote: Ingo Molnar writes: Yes, there's (obviously) compatibility requirements and artifacts and past mistakes (as with any software interface), but you need to admit it to Yes that's exactly what I meant. And we do make plenty of mistakes. And when we fix them, we have to maintain bug-compatibility to allow live migration from the broken version to the good version. If you're ever feeling overly happy, do some compat work in qemu and it will suck a year's worth or two of your life force a pop. yourself that your "virtualization is sloppy just like hardware" claim is just In my experience hardware is a lot less sloppy than software. Imagine your latest CPU had as many regressions as 2.6.32 @) I wish software and even VMs were as good. Me too. a cheap excuse to not do a proper job of interface engineering. Past mistakes cannot be easily fixed. And undoubtedly even the new shiny interfaces will have bugs and problems. Also the behaviour is often not completely understood. Maybe it can be easier debugged with fully available source, but even then it's hard to fix the old software (or rather even if you can fix it deploy the fixes). In that regard it's a lot like hardware. I agree with you that this makes it important to design good interfaces, but again realistically mistakes will be made and they cannot be all fixed retroactively. Our principal tool for this is to avoid introducing new interfaces whenever possible. We try to stick to established hardware standards (so we don't need to sloppily define them, and get guest compatibility for free). Hardware (both virt and non-virt) faces the same problems as software here. So as hardware solutions are introduced, we adopt them, and usually the virt extensions vendors follow with accelerations for these paths as well. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 06:44 PM, Gregory Haskins wrote: - Are a pure software concept By design. In fact, I would describe it as "software to software optimized" as opposed to trying to shoehorn into something that was designed as a software-to-hardware interface (and therefore has assumptions about the constraints in that environment that are not applicable in software-only). And that's the biggest mistake you can make. Look at Xen, for instance. The paravirtualized the fork out of everything that moved in order to get x86 virt going. And where are they now? x86_64 syscalls are slow since they have to trap to the hypervisor and (partially) flush the tlb. With npt or ept capable hosts performance is better for many workloads on fullvirt. And paravirt doesn't support Windows. Their unsung hero Jeremy is still trying to upstream dom0 Xen support. And they get to support it forever. VMware stuck with the hardware defined interfaces. Sure they had to implement binary translation to get there, but as a result, they only have to support one interface, all guests support it, and they can drop it on newer hosts where it doesn't give them anything. We had the advantage of course of starting with virt extensions, so it was a no-brainer: paravirt only where absolutely required. Where we deviated from this, it backfired. - Gregory claims that the AlacricityVM patches are not a replacement for KVM. I.e. there's no intention to phase out any 'old stuff' There's no reason to phase anything out, except perhaps the virtio-pci transport. This is one more transport, plugging into virtio underneath (just like virtio-pci, virtio-lguest, and virtio-s390). I am not even suggesting that the old transport has to go away, per se. It is the KVM maintainers who insist on it being all or nothing. For me, I do not see the big deal in having one more "model" option in the qemu cmd-line, but that is just my opinion. If the maintainers were really so adamant that choice is pure evil, I am not sure why we don't see patches for removing everything but one model type in each IO category. But I digress. We have to support users (also known as customers in some areas), so we have to keep the old stuff. We have limited resources, so we want to maintain as little as possible. We'll do it if we have to, but I'm totally unconvinced we have to. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 10:36 PM, Avi Kivity wrote: On 12/23/2009 06:44 PM, Gregory Haskins wrote: - Are a pure software concept By design. In fact, I would describe it as "software to software optimized" as opposed to trying to shoehorn into something that was designed as a software-to-hardware interface (and therefore has assumptions about the constraints in that environment that are not applicable in software-only). And that's the biggest mistake you can make. Look at Xen, for instance. The paravirtualized the fork out of everything that moved in order to get x86 virt going. And where are they now? x86_64 syscalls are slow since they have to trap to the hypervisor and (partially) flush the tlb. With npt or ept capable hosts performance is better for many workloads on fullvirt. And paravirt doesn't support Windows. Their unsung hero Jeremy is still trying to upstream dom0 Xen support. And they get to support it forever. VMware stuck with the hardware defined interfaces. Sure they had to implement binary translation to get there, but as a result, they only have to support one interface, all guests support it, and they can drop it on newer hosts where it doesn't give them anything. As a twist on this, the VMware paravirt driver interface is so hardware-like that they're getting hardware vendors to supply cards that implement it. Try that with a pure software approach. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 11:29 AM, Linus Torvalds wrote: On Wed, 23 Dec 2009, Gregory Haskins wrote: And upstream submission is not always like this! I would think the process would come to a grinding halt if it were ;) Well, in all honesty, if it had been non-virtualized drivers I would just have pulled. The pull request all looked sane, the diffstat looked clean and non-intrusive, and I had no problems with any of that. But the virtualization people always argue about the fifty-eleven different ways of doing things, and unlike real drivers - where the actual hardware places constraints on what the heck is going on - virtualization people seem to revel in making new interfaces weekly, and tend to be only incidentally limited by hardware (ie hardware interfaces may limit some _details_, but seldom any higher-level arguments). So when I see another virtualization interface, I want the virtualization people to just argue it out amongst themselves. Actually, this sentiment is really the basis of this whole discussion. KVM is the product of learnign the hard way that that inventing interfaces just because we can is a total waste of time. Our current I/O infrastructure is based on PCI devices that we can emulate efficiently. They look, feel, and taste like real hardware devices. We try to be as boring as humanly possible and so far, it's worked out extremely well for us. Thanks to the virtue of me personally not caring one whit about virtualization, I can stand back and just watch the fireworks. That's ashame, because I wish more people with a practical sentiment cared about virtualization to discourage the general silliness that seems to be all too common in this space. Regards, Anthony LIguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 07:51:50PM +0100, Ingo Molnar wrote: > > * Andi Kleen wrote: > > > > - Are a pure software concept and any compatibility mismatch is > > > > > > self-inflicted. The patches are in fact breaking the ABI to KVM > > > > > > > In practice, especially considering older kernel releases, VMs behave like > > hardware, with all its quirks, compatibility requirements, sometimes not > > fully understood, etc. > > I stopped reading your reply here. That's not actually fully true of KVM, at > all. > > Virtualization isnt voodoo magic with some hidden souce in some magic > hardware > component that no-one can understand fully. This isnt some mystic hardware > vendor coming up with some code and going away in the next quarter, with > barely anything documented and thousands of users left with hardware > components which we need to support under Linux somehow. > > This is Linux virtualization, where _both_ the host and the guest source code > is fully known, and bugs (if any) can be found with a high degree of It may sound strange but Windows is very popular guest and last I checked my HW there was no Windows sources there, but the answer to that is to emulate HW as close as possible to real one and then closed source guests will not have a reason to be upset. > determinism. This is Linux where the players dont just vanish overnight, and > are expected to do a proper job. > > Yes, there's (obviously) compatibility requirements and artifacts and past > mistakes (as with any software interface), but you need to admit it to > yourself that your "virtualization is sloppy just like hardware" claim is > just > a cheap excuse to not do a proper job of interface engineering. > > Thanks, > > Ingo > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 3:36 PM, Avi Kivity wrote: > On 12/23/2009 06:44 PM, Gregory Haskins wrote: >> >>> - Are a pure software concept >>> >> By design. In fact, I would describe it as "software to software >> optimized" as opposed to trying to shoehorn into something that was >> designed as a software-to-hardware interface (and therefore has >> assumptions about the constraints in that environment that are not >> applicable in software-only). >> >> > > And that's the biggest mistake you can make. Sorry, that is just wrong or you wouldn't have virtio either. > Look at Xen, for > instance. The paravirtualized the fork out of everything that moved in > order to get x86 virt going. And where are they now? x86_64 syscalls > are slow since they have to trap to the hypervisor and (partially) flush > the tlb. With npt or ept capable hosts performance is better for many > workloads on fullvirt. And paravirt doesn't support Windows. Their > unsung hero Jeremy is still trying to upstream dom0 Xen support. And > they get to support it forever. We are only talking about PV-IO here, so not apples to apples to what Xen is going through. > > VMware stuck with the hardware defined interfaces. Sure they had to > implement binary translation to get there, but as a result, they only > have to support one interface, all guests support it, and they can drop > it on newer hosts where it doesn't give them anything. Again, you are confusing PV-IO. Not relevant here. Afaict, vmware, kvm, xen, etc, all still do PV-IO and likely will for the foreseeable future. -Greg signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 4:01 PM, Avi Kivity wrote: > On 12/23/2009 10:36 PM, Avi Kivity wrote: >> On 12/23/2009 06:44 PM, Gregory Haskins wrote: >>> - Are a pure software concept >>> By design. In fact, I would describe it as "software to software >>> optimized" as opposed to trying to shoehorn into something that was >>> designed as a software-to-hardware interface (and therefore has >>> assumptions about the constraints in that environment that are not >>> applicable in software-only). >>> >> >> And that's the biggest mistake you can make. Look at Xen, for >> instance. The paravirtualized the fork out of everything that moved >> in order to get x86 virt going. And where are they now? x86_64 >> syscalls are slow since they have to trap to the hypervisor and >> (partially) flush the tlb. With npt or ept capable hosts performance >> is better for many workloads on fullvirt. And paravirt doesn't >> support Windows. Their unsung hero Jeremy is still trying to upstream >> dom0 Xen support. And they get to support it forever. >> >> VMware stuck with the hardware defined interfaces. Sure they had to >> implement binary translation to get there, but as a result, they only >> have to support one interface, all guests support it, and they can >> drop it on newer hosts where it doesn't give them anything. > > As a twist on this, the VMware paravirt driver interface is so > hardware-like that they're getting hardware vendors to supply cards that > implement it. Try that with a pure software approach. Any hardware engineer (myself included) will tell you that, generally speaking, what you can do in hardware you can do in software (think of what QEMU does today, for instance). It's purely a cost/performance tradeoff. I can at least tell you that is true of vbus. Anything on the vbus side would be equally eligible for a hardware implementation, though there is not reason to do this today since we have equivalent functionality in baremetal already. The only motiviation is if you wanted to preserve ABI etc, which is what vmware is presumably after. However, I am not advocating this as necessary at this juncture. So sorry, your statement is not relevant. -Greg > signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 11:28:08AM -0800, Ira W. Snyder wrote: > On Wed, Dec 23, 2009 at 12:34:44PM -0500, Gregory Haskins wrote: > > On 12/23/09 1:15 AM, Kyle Moffett wrote: > > > On Tue, Dec 22, 2009 at 12:36, Gregory Haskins > > > wrote: > > >> On 12/22/09 2:57 AM, Ingo Molnar wrote: > > >>> * Gregory Haskins wrote: > > Actually, these patches have nothing to do with the KVM folks. [...] > > >>> > > >>> That claim is curious to me - the AlacrityVM host > > >> > > >> It's quite simple, really. These drivers support accessing vbus, and > > >> vbus is hypervisor agnostic. In fact, vbus isn't necessarily even > > >> hypervisor related. It may be used anywhere where a Linux kernel is the > > >> "io backend", which includes hypervisors like AlacrityVM, but also > > >> userspace apps, and interconnected physical systems as well. So focus on interconnecting physical systems I think would be one way for vbus to stop conflicting with KVM. If drivers for such systems appear I expect that relevant (hypervisor-agnostic) vbus bits would be very uncontroversial. This would not be the first technology to make the jump from attempting to be a PCI replacement to being an interconnect btw, I think infiniband did this as well. -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
> > This is Linux virtualization, where _both_ the host and the guest source > > code > > is fully known, and bugs (if any) can be found with a high degree of > It may sound strange but Windows is very popular guest and last I > checked my HW there was no Windows sources there, but the answer to that > is to emulate HW as close as possible to real one and then closed source > guests will not have a reason to be upset. > > > determinism. This is Linux where the players dont just vanish overnight, > > and > > are expected to do a proper job. And without even getting into closed/proprietary guests, virt is useful for testing/developing/deploying many free OSes, eg FreeBSD, NetBSD, OpenBSD, Hurd, , etc. Not to mention just wanting a stable [virtual] platform to run on. So having a virtual platform whose interface doesn't change very often or very much has a lot of value at least in avoiding churn in guest OSes. - R. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/24/2009 11:31 AM, Gregory Haskins wrote: On 12/23/09 3:36 PM, Avi Kivity wrote: On 12/23/2009 06:44 PM, Gregory Haskins wrote: - Are a pure software concept By design. In fact, I would describe it as "software to software optimized" as opposed to trying to shoehorn into something that was designed as a software-to-hardware interface (and therefore has assumptions about the constraints in that environment that are not applicable in software-only). And that's the biggest mistake you can make. Sorry, that is just wrong or you wouldn't have virtio either. Things are not black and white. I prefer not to have paravirtualization at all. When there is no alternative, I prefer to limit it to the device level and keep it off the bus level. Look at Xen, for instance. The paravirtualized the fork out of everything that moved in order to get x86 virt going. And where are they now? x86_64 syscalls are slow since they have to trap to the hypervisor and (partially) flush the tlb. With npt or ept capable hosts performance is better for many workloads on fullvirt. And paravirt doesn't support Windows. Their unsung hero Jeremy is still trying to upstream dom0 Xen support. And they get to support it forever. We are only talking about PV-IO here, so not apples to apples to what Xen is going through. The same principles apply. VMware stuck with the hardware defined interfaces. Sure they had to implement binary translation to get there, but as a result, they only have to support one interface, all guests support it, and they can drop it on newer hosts where it doesn't give them anything. Again, you are confusing PV-IO. Not relevant here. Afaict, vmware, kvm, xen, etc, all still do PV-IO and likely will for the foreseeable future. They're all doing it very differently: - pure emulation (qemu e1000, etc.) - pci device (vmware, virtio/pci) - paravirt bus bridged through a pci device (Xen hvm, Hyper-V (I think), venet/vbus) - paravirt bus (Xen pv, early vbus, virtio/lguest, virtio/s390) The higher you are up this scale the easier things are, so once you get reasonable performance there is no need to descend further. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/24/2009 11:36 AM, Gregory Haskins wrote: As a twist on this, the VMware paravirt driver interface is so hardware-like that they're getting hardware vendors to supply cards that implement it. Try that with a pure software approach. Any hardware engineer (myself included) will tell you that, generally speaking, what you can do in hardware you can do in software (think of what QEMU does today, for instance). It's purely a cost/performance tradeoff. I can at least tell you that is true of vbus. Anything on the vbus side would be equally eligible for a hardware implementation, though there is not reason to do this today since we have equivalent functionality in baremetal already. There's a huge difference in the probability of vmware getting cards to their spec, or x86 vendors improving interrupt delivery to guests, compared to vbus being implemented in hardware. The only motiviation is if you wanted to preserve ABI etc, which is what vmware is presumably after. However, I am not advocating this as necessary at this juncture. Maybe AlacrityVM users don't care about compatibility, but my users do. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/27/09 4:33 AM, Avi Kivity wrote: > On 12/24/2009 11:36 AM, Gregory Haskins wrote: >>> As a twist on this, the VMware paravirt driver interface is so >>> hardware-like that they're getting hardware vendors to supply cards that >>> implement it. Try that with a pure software approach. >>> >> Any hardware engineer (myself included) will tell you that, generally >> speaking, what you can do in hardware you can do in software (think of >> what QEMU does today, for instance). It's purely a cost/performance >> tradeoff. >> >> I can at least tell you that is true of vbus. Anything on the vbus side >> would be equally eligible for a hardware implementation, though there is >> not reason to do this today since we have equivalent functionality in >> baremetal already. > > There's a huge difference in the probability of vmware getting cards to > their spec, or x86 vendors improving interrupt delivery to guests, > compared to vbus being implemented in hardware. Thats not relevant, however. I said in the original quote that you snipped that I made it a software design on purpose, and you tried to somehow paint that as a negative because vmware made theirs "hardware-like" and you implied it could not be done with my approach with the statement "try that with a pure software approach". And the bottom line is that the statement is incorrect and/or misleading. > >> The only motiviation is if you wanted to preserve >> ABI etc, which is what vmware is presumably after. However, I am not >> advocating this as necessary at this juncture. >> > > Maybe AlacrityVM users don't care about compatibility, but my users do. Again, not relevant to this thread. Making your interface "hardware-like" buys you nothing in the end, as you ultimately need to load drivers in the guest either way, and any major OS lets you extend both devices and buses with relative ease. The only counter example would be if you truly were "hardware-exactly" like e1000 emulation, but we already know that this means it is hardware centric and not "exit-rate aware" and would perform poorly. Otherwise "compatible" is purely a point on the time line (for instance, the moment virtio-pci ABI shipped), not an architectural description such as "hardware-like". > signature.asc Description: OpenPGP digital signature
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/27/2009 03:34 PM, Gregory Haskins wrote: On 12/27/09 4:33 AM, Avi Kivity wrote: On 12/24/2009 11:36 AM, Gregory Haskins wrote: As a twist on this, the VMware paravirt driver interface is so hardware-like that they're getting hardware vendors to supply cards that implement it. Try that with a pure software approach. Any hardware engineer (myself included) will tell you that, generally speaking, what you can do in hardware you can do in software (think of what QEMU does today, for instance). It's purely a cost/performance tradeoff. I can at least tell you that is true of vbus. Anything on the vbus side would be equally eligible for a hardware implementation, though there is not reason to do this today since we have equivalent functionality in baremetal already. There's a huge difference in the probability of vmware getting cards to their spec, or x86 vendors improving interrupt delivery to guests, compared to vbus being implemented in hardware. Thats not relevant, however. I said in the original quote that you snipped that I made it a software design on purpose, and you tried to somehow paint that as a negative because vmware made theirs "hardware-like" and you implied it could not be done with my approach with the statement "try that with a pure software approach". And the bottom line is that the statement is incorrect and/or misleading. It's not incorrect. VMware stuck to the pci specs and as a result they can have hardware implement their virtual NIC protocol. For vbus this is much harder to do since you need a side-channel between different cards to coordinate interrupt delivery. In theory you can do eveything if you don't consider practicalities. That's a digression, though, I'm not suggesting we'll see virtio hardware or that this is a virtio/pci advantage vs. vbus. It's an anecdote showing that sticking with specs has its advantages. wrt pci vs vbus, the difference is in the ability to use improvements in interrupt delivery accelerations in virt hardware. If this happens, virtio/pci can immediately take advantage of it, while vbus has to stick with software delivery for backward compatibility, and all that code becomes a useless support burden. As an example of what hardware can do when it really sets its mind to it, s390 can IPI from vcpu to vcpu without exiting to the host. The only motiviation is if you wanted to preserve ABI etc, which is what vmware is presumably after. However, I am not advocating this as necessary at this juncture. Maybe AlacrityVM users don't care about compatibility, but my users do. Again, not relevant to this thread. Making your interface "hardware-like" buys you nothing in the end, as you ultimately need to load drivers in the guest either way, and any major OS lets you extend both devices and buses with relative ease. The only counter example would be if you truly were "hardware-exactly" like e1000 emulation, but we already know that this means it is hardware centric and not "exit-rate aware" and would perform poorly. Otherwise "compatible" is purely a point on the time line (for instance, the moment virtio-pci ABI shipped), not an architectural description such as "hardware-like". True, not related to the thread. But it is a problem. The difference between virtio and vbus here is that virtio is already deployed and its users expect not to reinstall drivers [1]. Before virtio existed, people could not deploy performance sensitive applications on kvm. Now that it exists, we have to support it without requiring users to touch their guests. That means that without proof that virtio cannot be scaled, we'll keep supporting and extending it. [1] Another difference is the requirement for writing a "bus driver" for every supported guest, which means dealing with icky bits like hotplug. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/27/09 8:49 AM, Avi Kivity wrote: > On 12/27/2009 03:34 PM, Gregory Haskins wrote: >> On 12/27/09 4:33 AM, Avi Kivity wrote: >> >>> On 12/24/2009 11:36 AM, Gregory Haskins wrote: >>> > As a twist on this, the VMware paravirt driver interface is so > hardware-like that they're getting hardware vendors to supply cards > that > implement it. Try that with a pure software approach. > > Any hardware engineer (myself included) will tell you that, generally speaking, what you can do in hardware you can do in software (think of what QEMU does today, for instance). It's purely a cost/performance tradeoff. I can at least tell you that is true of vbus. Anything on the vbus side would be equally eligible for a hardware implementation, though there is not reason to do this today since we have equivalent functionality in baremetal already. >>> There's a huge difference in the probability of vmware getting cards to >>> their spec, or x86 vendors improving interrupt delivery to guests, >>> compared to vbus being implemented in hardware. >>> >> Thats not relevant, however. I said in the original quote that you >> snipped that I made it a software design on purpose, and you tried to >> somehow paint that as a negative because vmware made theirs >> "hardware-like" and you implied it could not be done with my approach >> with the statement "try that with a pure software approach". And the >> bottom line is that the statement is incorrect and/or misleading. >> > > It's not incorrect. At the very best it's misleading. > VMware stuck to the pci specs and as a result they > can have hardware implement their virtual NIC protocol. For vbus this > is much harder Not really. > to do since you need a side-channel between different > cards to coordinate interrupt delivery. In theory you can do eveything > if you don't consider practicalities. pci based designs, such as vmware and virtio-pci arent free of this notion either. They simply rely on APIC emulation for the irq-chip, and it just so happens that vbus implements a different irq-chip (more specifically, the connector that we employ between the guest and vbus does). On one hand, you have the advantage of the guest already supporting the irq-chip ABI, and on other other, you have an optimized (e.g. shared memory based inject/ack) and feature enhanced ABI (interrupt priority, no IDT constraints, etc). The are pros and cons to either direction, but the vbus project charter is to go for maximum performance and features, so that is acceptable to us. > > That's a digression, though, I'm not suggesting we'll see virtio > hardware or that this is a virtio/pci advantage vs. vbus. It's an > anecdote showing that sticking with specs has its advantages. It also has distinct disadvantages. For instance, the PCI spec is gigantic, yet almost none of it is needed to do the job here. When you are talking full-virt, you are left with no choice. With para-virt, you do have a choice, and the vbus-connector for AlacrityVM capitalizes on this. As an example, think about all the work that went into emulating the PCI chipset, the APIC chipset, MSI-X support, irq-routing, etc, when all you needed was a simple event-queue to indicate that an event (e.g. an "interrupt") occurred. This is an example connector in vbus: http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=blob;f=kernel/vbus/connectors/null.c;h=b6d16cb68b7e49e07528278bc9f5b73e1dac0c2f;hb=HEAD It encapsulates all of hotplug, signal (interrupt) routing, and memory routing for both sides of the "link" in 584 lines of code. And that also implicitly brings in device discovery and configuration since that is covered by the vbus framework. Try doing that with PCI, especially when you are not already under the qemu umbrella, and the "standards based" approach suddenly doesn't look very attractive. > > wrt pci vs vbus, the difference is in the ability to use improvements in > interrupt delivery accelerations in virt hardware. Most of which will also apply to the current vbus design as well since at some point I have to have an underlying IDT mechanism too, btw. > If this happens, > virtio/pci can immediately take advantage of it, while vbus has to stick > with software delivery for backward compatibility, and all that code > becomes a useless support burden. > The shared-memory path will always be the fastest anyway, so I am not too worried about it. But vbus supports feature negotiation, so we can always phase that out if need be, same as anything else. > As an example of what hardware can do when it really sets its mind to > it, s390 can IPI from vcpu to vcpu without exiting to the host. Great! I am just not in the practice of waiting for hardware to cover sloppy software. There is a ton of impracticality in doing so, such as the fact that the hardware, even once
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, 2009-12-23 at 13:14 +0100, Andi Kleen wrote: > > http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf > > > > See slide 32. This is without vhost-net. > > Thanks. Do you also have latency numbers? > > It seems like there's definitely still potential for improvement > with messages <4K. But for the large messages they indeed > look rather good. > > It's unclear what message size the Alacrity numbers used, but I presume > it was rather large. > No. It was 1500. Please see: http://developer.novell.com/wiki/index.php/AlacrityVM/Results Best, -PWM > -Andi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 5:22 AM, Avi Kivity wrote: > > There was no attempt by Gregory to improve virtio-net. If you truly do not understand why your statement is utterly wrong at this point in the discussion, I feel sorry for you. If you are trying to be purposely disingenuous, you should be ashamed of yourself. In any case, your statement is demonstrably bogus, but you should already know this given that we talked about at least several times. To refresh your memory: http://patchwork.kernel.org/patch/17428/ In case its not blatantly clear, which I would hope it would be to anyone that understands the problem space: What that patch would do is allow an unmodified virtio-net to bridge to a vbus based virtio-net backend. (Also note that this predates vhost-net by months (the date in that thread is 4/9/2009) in case you are next going to try to argue that it does nothing over vhost-net). This would mean that virtio-net would gain most of the benefits I have been advocating (fewer exits, cheaper exits, concurrent execution, etc). So this would very much improve virtio-net indeed, given how poorly the current backend was performing. I tried to convince the team to help me build it out to completion on multiple occasions, but that request was answered with "sorry, we are doing our own thing instead". You can say that you didn't like my approach, since that is a subjective opinion. But to say that I didn't attempt to improve it is a flat out wrong, and I do not appreciate it. -Greg signature.asc Description: OpenPGP digital signature
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/09 12:52 PM, Peter W. Morreale wrote: > On Wed, 2009-12-23 at 13:14 +0100, Andi Kleen wrote: >>> http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf >>> >>> See slide 32. This is without vhost-net. >> >> Thanks. Do you also have latency numbers? >> >> It seems like there's definitely still potential for improvement >> with messages <4K. But for the large messages they indeed >> look rather good. >> >> It's unclear what message size the Alacrity numbers used, but I presume >> it was rather large. >> > > No. It was 1500. Please see: > > http://developer.novell.com/wiki/index.php/AlacrityVM/Results > Note: 1500 was the L2 MTU, not necessarily the L3/L4 size which was probably much larger (though I do not recall what exactly atm). -Greg signature.asc Description: OpenPGP digital signature
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
* Peter W. Morreale (pmorre...@novell.com) wrote: > On Wed, 2009-12-23 at 13:14 +0100, Andi Kleen wrote: > > > http://www.redhat.com/f/pdf/summit/cwright_11_open_source_virt.pdf > > > > > > See slide 32. This is without vhost-net. > > > > Thanks. Do you also have latency numbers? > > > > It seems like there's definitely still potential for improvement > > with messages <4K. But for the large messages they indeed > > look rather good. > > > > It's unclear what message size the Alacrity numbers used, but I presume > > it was rather large. > > No. It was 1500. Please see: > > http://developer.novell.com/wiki/index.php/AlacrityVM/Results That's just MTU. Not the message size. We can infer the message size by the bare metal results (reasonably large), but is helpful to record that. thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 09:09:21AM -0600, Anthony Liguori wrote: > On 12/23/2009 12:15 AM, Kyle Moffett wrote: > > This is actually something that is of particular interest to me. I > > have a few prototype boards right now with programmable PCI-E > > host/device links on them; one of my long-term plans is to finagle > > vbus into providing multiple "virtual" devices across that single > > PCI-E interface. > > > > Specifically, I want to be able to provide virtual NIC(s), serial > > ports and serial consoles, virtual block storage, and possibly other > > kinds of interfaces. My big problem with existing virtio right now > > (although I would be happy to be proven wrong) is that it seems to > > need some sort of out-of-band communication channel for setting up > > devices, not to mention it seems to need one PCI device per virtual > > device. > > We've been thinking about doing a virtio-over-IP mechanism such that you > could remote the entire virtio bus to a separate physical machine. > virtio-over-IB is probably more interesting since you can make use of > RDMA. virtio-over-PCI-e would work just as well. > I didn't know you were interested in this as well. See my later reply to Kyle for a lot of code that I've written with this in mind. > virtio is a layered architecture. Device enumeration/discovery sits at > a lower level than the actual device ABIs. The device ABIs are > implemented on top of a bulk data transfer API. The reason for this > layering is so that we can reuse PCI as an enumeration/discovery > mechanism. This tremendenously simplifies porting drivers to other OSes > and let's us use PCI hotplug automatically. We get integration into all > the fancy userspace hotplug support for free. > > But both virtio-lguest and virtio-s390 use in-band enumeration and > discovery since they do not have support for PCI on either platform. > I'm interested in the same thing, just over PCI. The only PCI agent systems I've used are not capable of manipulating the PCI configuration space in such a way that virtio-pci is usable on them. This means creating your own enumeration mechanism. Which sucks. See my virtio-phys code (http://www.mmarray.org/~iws/virtio-phys/) for an example of how I did it. It was modeled on lguest. Help is appreciated. Ira -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 08:15 PM, Gregory Haskins wrote: On 12/23/09 5:22 AM, Avi Kivity wrote: There was no attempt by Gregory to improve virtio-net. If you truly do not understand why your statement is utterly wrong at this point in the discussion, I feel sorry for you. If you are trying to be purposely disingenuous, you should be ashamed of yourself. In any case, your statement is demonstrably bogus, but you should already know this given that we talked about at least several times. There's no need to feel sorry for me, thanks. There's no reason for me to be ashamed, either. And there's no need to take the discussion to personal levels. Please keep it technical. To refresh your memory: http://patchwork.kernel.org/patch/17428/ This is not an attempt to improve virtio-net, it's an attempt to push vbus. With this, virtio-net doesn't become any faster, since the greatest bottleneck is not removed, it remains in userspace. If you wanted to improve virtio-net, you would port venet-host to the virtio-net guest/host interface, and port any secret sauce in venet(-guest) to virtio-net. After that we could judge what vbus' contribution to the equation is. In case its not blatantly clear, which I would hope it would be to anyone that understands the problem space: What that patch would do is allow an unmodified virtio-net to bridge to a vbus based virtio-net backend. (Also note that this predates vhost-net by months (the date in that thread is 4/9/2009) in case you are next going to try to argue that it does nothing over vhost-net). Without the backend, it is useless. It demonstrates vbus' flexibility quite well, but does nothing for virtio-net or its users, at least without a lot more work. This would mean that virtio-net would gain most of the benefits I have been advocating (fewer exits, cheaper exits, concurrent execution, etc). So this would very much improve virtio-net indeed, given how poorly the current backend was performing. I tried to convince the team to help me build it out to completion on multiple occasions, but that request was answered with "sorry, we are doing our own thing instead". You can say that you didn't like my approach, since that is a subjective opinion. But to say that I didn't attempt to improve it is a flat out wrong, and I do not appreciate it. Cutting down on the rhetoric is more important than cutting down exits at this point in time. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
(Sorry for top post...on a mobile) When someone repeatedly makes a claim you believe to be wrong and you correct them, you start to wonder if that person has a less than honorable agenda. In any case, I overreacted. For that, I apologize. That said, you are still incorrect. With what I proposed, the model will run as an in-kernel vbus device, and no longer run in userspace. It would therefore improve virtio-net as I stated, much in the same way vhost-net or venet-tap do today. FYI I am about to log out for the long holiday, so will be unresponsive for a bit. Kind Regards, -Greg On 12/23/09, Avi Kivity wrote: > On 12/23/2009 08:15 PM, Gregory Haskins wrote: >> On 12/23/09 5:22 AM, Avi Kivity wrote: >> >> >>> There was no attempt by Gregory to improve virtio-net. >>> >> If you truly do not understand why your statement is utterly wrong at >> this point in the discussion, I feel sorry for you. If you are trying >> to be purposely disingenuous, you should be ashamed of yourself. In any >> case, your statement is demonstrably bogus, but you should already know >> this given that we talked about at least several times. >> > > There's no need to feel sorry for me, thanks. There's no reason for me > to be ashamed, either. And there's no need to take the discussion to > personal levels. Please keep it technical. > > >> To refresh your memory: http://patchwork.kernel.org/patch/17428/ >> > > This is not an attempt to improve virtio-net, it's an attempt to push > vbus. With this, virtio-net doesn't become any faster, since the > greatest bottleneck is not removed, it remains in userspace. > > If you wanted to improve virtio-net, you would port venet-host to the > virtio-net guest/host interface, and port any secret sauce in > venet(-guest) to virtio-net. After that we could judge what vbus' > contribution to the equation is. > >> In case its not blatantly clear, which I would hope it would be to >> anyone that understands the problem space: What that patch would do is >> allow an unmodified virtio-net to bridge to a vbus based virtio-net >> backend. (Also note that this predates vhost-net by months (the date in >> that thread is 4/9/2009) in case you are next going to try to argue that >> it does nothing over vhost-net). >> > > Without the backend, it is useless. It demonstrates vbus' flexibility > quite well, but does nothing for virtio-net or its users, at least > without a lot more work. > >> This would mean that virtio-net would gain most of the benefits I have >> been advocating (fewer exits, cheaper exits, concurrent execution, etc). >> So this would very much improve virtio-net indeed, given how poorly the >> current backend was performing. I tried to convince the team to help me >> build it out to completion on multiple occasions, but that request was >> answered with "sorry, we are doing our own thing instead". You can say >> that you didn't like my approach, since that is a subjective opinion. >> But to say that I didn't attempt to improve it is a flat out wrong, and >> I do not appreciate it. >> > > Cutting down on the rhetoric is more important than cutting down exits > at this point in time. > > -- > I have a truly marvellous patch that fixes the bug which this > signature is too narrow to contain. > > -- Sent from my mobile device -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 01:54 PM, Ira W. Snyder wrote: On Wed, Dec 23, 2009 at 09:09:21AM -0600, Anthony Liguori wrote: I didn't know you were interested in this as well. See my later reply to Kyle for a lot of code that I've written with this in mind. BTW, in the future, please CC me or CC virtualizat...@lists.linux-foundation.org. Or certainly k...@vger. I never looked at the virtio-over-pci patchset although I've heard it referenced before. But both virtio-lguest and virtio-s390 use in-band enumeration and discovery since they do not have support for PCI on either platform. I'm interested in the same thing, just over PCI. The only PCI agent systems I've used are not capable of manipulating the PCI configuration space in such a way that virtio-pci is usable on them. virtio-pci is the wrong place to start if you want to use a PCI *device* as the virtio bus. virtio-pci is meant to use the PCI bus as the virtio bus. That's a very important requirement for us because it maintains the relationship of each device looking like a normal PCI device. This means creating your own enumeration mechanism. Which sucks. I don't think it sucks. The idea is that we don't want to unnecessarily reinvent things. Of course, the key feature of virtio is that it makes it possible for you to create your own enumeration mechanism if you're so inclined. See my virtio-phys code (http://www.mmarray.org/~iws/virtio-phys/) for an example of how I did it. It was modeled on lguest. Help is appreciated. If it were me, I'd take a much different approach. I would use a very simple device with a single transmit and receive queue. I'd create a standard header, and the implement a command protocol on top of it. You'll be able to support zero copy I/O (although you'll have a fixed number of outstanding requests). You would need a single large ring. But then again, I have no idea what your requirements are. You could probably get far treating the thing as a network device and just doing ATAoE or something like that. Regards, Anthony Liguori Ira -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 04:58:37PM -0600, Anthony Liguori wrote: > On 12/23/2009 01:54 PM, Ira W. Snyder wrote: > > On Wed, Dec 23, 2009 at 09:09:21AM -0600, Anthony Liguori wrote: > > > I didn't know you were interested in this as well. See my later reply to > > Kyle for a lot of code that I've written with this in mind. > > > BTW, in the future, please CC me or CC > virtualizat...@lists.linux-foundation.org. Or certainly k...@vger. I > never looked at the virtio-over-pci patchset although I've heard it > referenced before. > Will do. I wouldn't think k...@vger would be on-topic. I'm not interested in KVM (though I do use it constantly, it is great). I'm only interested in using virtio as a transport between physical systems. Is it a place where discussing virtio by itself is on-topic? > >> But both virtio-lguest and virtio-s390 use in-band enumeration and > >> discovery since they do not have support for PCI on either platform. > >> > > > > I'm interested in the same thing, just over PCI. The only PCI agent > > systems I've used are not capable of manipulating the PCI configuration > > space in such a way that virtio-pci is usable on them. > > virtio-pci is the wrong place to start if you want to use a PCI *device* > as the virtio bus. virtio-pci is meant to use the PCI bus as the virtio > bus. That's a very important requirement for us because it maintains > the relationship of each device looking like a normal PCI device. > > > This means > > creating your own enumeration mechanism. Which sucks. > > I don't think it sucks. The idea is that we don't want to unnecessarily > reinvent things. > > Of course, the key feature of virtio is that it makes it possible for > you to create your own enumeration mechanism if you're so inclined. > > > See my virtio-phys > > code (http://www.mmarray.org/~iws/virtio-phys/) for an example of how I > > did it. It was modeled on lguest. Help is appreciated. > > If it were me, I'd take a much different approach. I would use a very > simple device with a single transmit and receive queue. I'd create a > standard header, and the implement a command protocol on top of it. > You'll be able to support zero copy I/O (although you'll have a fixed > number of outstanding requests). You would need a single large ring. > > But then again, I have no idea what your requirements are. You could > probably get far treating the thing as a network device and just doing > ATAoE or something like that. > I've got a single PCI Host (master) with ~20 PCI slots. Physically, it is a backplane in a cPCI chassis, but the form factor is irrelevant. It is regular PCI from a software perspective. Into this backplane, I plug up to 20 PCI Agents (slaves). They are powerpc computers, almost identical to the Freescale MPC8349EMDS board. They're full-featured powerpc computers, with CPU, RAM, etc. They can run standalone. I want to use the PCI backplane as a data transport. Specifically, I want to transport ethernet over the backplane, so I can have the powerpc boards mount their rootfs via NFS, etc. Everyone knows how to write network daemons. It is a good and very well known way to transport data between systems. On the PCI bus, the powerpc systems expose 3 PCI BAR's. The size is configureable, as is the memory location at which they point. What I cannot do is get notified when a read/write hits the BAR. There is a feature on the board which allows me to generate interrupts in either direction: agent->master (PCI INTX) and master->agent (via an MMIO register). The PCI vendor ID and device ID are not configureable. One thing I cannot assume is that the PCI master system is capable of performing DMA. In my system, it is a Pentium3 class x86 machine, which has no DMA engine. However, the PowerPC systems do have DMA engines. In virtio terms, it was suggested to make the powerpc systems the "virtio hosts" (running the backends) and make the x86 (PCI master) the "virtio guest" (running virtio-net, etc.). I'm not sure what you're suggesting in the paragraph above. I want to use virtio-net as the transport, I do not want to write my own virtual-network driver. Can you please clarify? Hopefully that explains what I'm trying to do. I'd love someone to help guide me in the right direction here. I want something to fill this need in mainline. I've been contacted seperately by 10+ people also looking for a similar solution. I hunch most of them end up doing what I did: write a quick-and-dirty network driver. I've been working on this for a year, just to give an idea. PS - should I create a new thread on the two mailing lists mentioned above? I don't want to go too far off-topic in an alacrityvm thread. :) Ira -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Wed, Dec 23, 2009 at 17:58, Anthony Liguori wrote: > On 12/23/2009 01:54 PM, Ira W. Snyder wrote: >> On Wed, Dec 23, 2009 at 09:09:21AM -0600, Anthony Liguori wrote: >>> But both virtio-lguest and virtio-s390 use in-band enumeration and >>> discovery since they do not have support for PCI on either platform. >> >> I'm interested in the same thing, just over PCI. The only PCI agent >> systems I've used are not capable of manipulating the PCI configuration >> space in such a way that virtio-pci is usable on them. > > virtio-pci is the wrong place to start if you want to use a PCI *device* as > the virtio bus. virtio-pci is meant to use the PCI bus as the virtio bus. > That's a very important requirement for us because it maintains the > relationship of each device looking like a normal PCI device. > >> This means >> creating your own enumeration mechanism. Which sucks. > > I don't think it sucks. The idea is that we don't want to unnecessarily > reinvent things. > > Of course, the key feature of virtio is that it makes it possible for you to > create your own enumeration mechanism if you're so inclined. See... the thing is... a lot of us random embedded board developers don't *want* to create our own enumeration mechanisms. I see a huge amount of value in vbus as a common zero-copy DMA-capable virtual-device interface, especially over miscellaneous non-PCI-bus interconnects. I mentioned my PCI-E boards earlier, but I would also personally be interested in using infiniband with RDMA as a virtual device bus. Basically, what it comes down to is vbus is practically useful as a generic way to provide a large number of hotpluggable virtual devices across an arbitrary interconnect. I agree that virtio works fine if you have some out-of-band enumeration and hotplug transport (like emulated PCI), but if you *don't* have that, it's pretty much faster to write your own set of paired network drivers than it is to write a whole enumeration and transport stack for virtio. On top of *that*, with the virtio approach I would need to write a whole bunch of tools to manage the set of virtual devices on my custom hardware. With vbus that management interface would be entirely common code across a potentially large number of virtualized physical transports. If vbus actually gets merged I will most likely be able to spend the time to get the PCI-E crosslinks on my boards talking vbus, otherwise it's liable to get completely shelved as "not worth the effort" to write all the glue to make virtio work. >> See my virtio-phys >> code (http://www.mmarray.org/~iws/virtio-phys/) for an example of how I >> did it. It was modeled on lguest. Help is appreciated. > > If it were me, I'd take a much different approach. I would use a very > simple device with a single transmit and receive queue. I'd create a > standard header, and the implement a command protocol on top of it. You'll > be able to support zero copy I/O (although you'll have a fixed number of > outstanding requests). You would need a single large ring. That's basically about as much work as writing entirely new network and serial drivers over PCI. Not only that, but I The beauty of vbus for me is that I could write a fairly simple logical-to-physical glue driver which lets vbus talk over my PCI-E or infiniband link and then I'm basically done. Not only that, but the tools for adding new virtual devices (ethernet, serial, block, etc) over vbus would be the same no matter what the underlying transport. > But then again, I have no idea what your requirements are. You could > probably get far treating the thing as a network device and just doing ATAoE > or something like that. Oh... yes... clearly the right solution is to forgo the whole zero-copy direct DMA of block writes and instead shuffle the whole thing into 16kB ATAoE packets. That would obviously be much faster on my little 1GHz PowerPC boards Sorry for the rant, but I really do think vbus is a valuable technology and it's a damn shame to see Gregory Haskins being put through this whole hassle. While most everybody else was griping about problems he sat down and wrote some very nice clean maintainable code to do what he needed. Not only that, but he designed a good enough model that it could be ported to run over almost everything from a single PCI-E link to an infiniband network. I personally would love to see vbus merged, into staging at the very least. I would definitely spend some time trying to make it work across PCI-E on my *very* *real* embedded boards. Look at vbus not as another virtualization ABI, but as a multiprotocol high-level device abstraction API that already has one well-implemented and high-performance user. Cheers, Kyle Moffett -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 10:52 PM, Kyle Moffett wrote: On Wed, Dec 23, 2009 at 17:58, Anthony Liguori wrote: Of course, the key feature of virtio is that it makes it possible for you to create your own enumeration mechanism if you're so inclined. See... the thing is... a lot of us random embedded board developers don't *want* to create our own enumeration mechanisms. I see a huge amount of value in vbus as a common zero-copy DMA-capable virtual-device interface, especially over miscellaneous non-PCI-bus interconnects. I mentioned my PCI-E boards earlier, but I would also personally be interested in using infiniband with RDMA as a virtual device bus. I understand what you're saying, but is there really a practical argument here? Infiniband already supports things like IPoIB and SCSI over IB. Is it necessary to add another layer on top of it? That said, it's easy enough to create a common enumeration mechanism for people to use with virtio. I doubt it's really that interesting but it's certainly quite reasonable. In fact, a lot of code could be reused from virtio-s390 or virtio-lguest. Basically, what it comes down to is vbus is practically useful as a generic way to provide a large number of hotpluggable virtual devices across an arbitrary interconnect. I agree that virtio works fine if you have some out-of-band enumeration and hotplug transport (like emulated PCI), but if you *don't* have that, it's pretty much faster to write your own set of paired network drivers than it is to write a whole enumeration and transport stack for virtio. On top of *that*, with the virtio approach I would need to write a whole bunch of tools to manage the set of virtual devices on my custom hardware. With vbus that management interface would be entirely common code across a potentially large number of virtualized physical transports. This particular use case really has nothing to do with virtualization. You really want an infiniband replacement using the PCI-e bus. There's so much on the horizon in this space that's being standardized in PCI-sig like MR-IOV. If it were me, I'd take a much different approach. I would use a very simple device with a single transmit and receive queue. I'd create a standard header, and the implement a command protocol on top of it. You'll be able to support zero copy I/O (although you'll have a fixed number of outstanding requests). You would need a single large ring. That's basically about as much work as writing entirely new network and serial drivers over PCI. Not only that, but I The beauty of vbus for me is that I could write a fairly simple logical-to-physical glue driver which lets vbus talk over my PCI-E or infiniband link and then I'm basically done. Is this something you expect people to use or is this a one-off project? I personally would love to see vbus merged, into staging at the very least. I would definitely spend some time trying to make it work across PCI-E on my *very* *real* embedded boards. Look at vbus not as another virtualization ABI, but as a multiprotocol high-level device abstraction API that already has one well-implemented and high-performance user. If someone wants to advocate vbus for non-virtualized purposes, I have no problem with that. I just don't think it makes sense in for KVM. virtio is not intended to be used for any possible purpose. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 05:42 PM, Ira W. Snyder wrote: I've got a single PCI Host (master) with ~20 PCI slots. Physically, it is a backplane in a cPCI chassis, but the form factor is irrelevant. It is regular PCI from a software perspective. Into this backplane, I plug up to 20 PCI Agents (slaves). They are powerpc computers, almost identical to the Freescale MPC8349EMDS board. They're full-featured powerpc computers, with CPU, RAM, etc. They can run standalone. I want to use the PCI backplane as a data transport. Specifically, I want to transport ethernet over the backplane, so I can have the powerpc boards mount their rootfs via NFS, etc. Everyone knows how to write network daemons. It is a good and very well known way to transport data between systems. On the PCI bus, the powerpc systems expose 3 PCI BAR's. The size is configureable, as is the memory location at which they point. What I cannot do is get notified when a read/write hits the BAR. There is a feature on the board which allows me to generate interrupts in either direction: agent->master (PCI INTX) and master->agent (via an MMIO register). The PCI vendor ID and device ID are not configureable. One thing I cannot assume is that the PCI master system is capable of performing DMA. In my system, it is a Pentium3 class x86 machine, which has no DMA engine. However, the PowerPC systems do have DMA engines. In virtio terms, it was suggested to make the powerpc systems the "virtio hosts" (running the backends) and make the x86 (PCI master) the "virtio guest" (running virtio-net, etc.). IMHO, virtio and vbus are both the wrong model for what you're doing. The key reason why is that virtio and vbus are generally designed around the concept that there is shared cache coherent memory from which you can use lock-less ring queues to implement efficient I/O. In your architecture, you do not have cache coherent shared memory. Instead, you have two systems connected via a PCI backplace with non-coherent shared memory. You probably need to use the shared memory as a bounce buffer and implement a driver on top of that. I'm not sure what you're suggesting in the paragraph above. I want to use virtio-net as the transport, I do not want to write my own virtual-network driver. Can you please clarify? virtio-net and vbus are going to be overly painful for you to use because no one end can access arbitrary memory in the other end. Hopefully that explains what I'm trying to do. I'd love someone to help guide me in the right direction here. I want something to fill this need in mainline. If I were you, I would write a custom network driver. virtio-net is awfully small (just a few hundred lines). I'd use that as a basis but I would not tie into virtio or vbus. The paradigms don't match. I've been contacted seperately by 10+ people also looking for a similar solution. I hunch most of them end up doing what I did: write a quick-and-dirty network driver. I've been working on this for a year, just to give an idea. The whole architecture of having multiple heterogenous systems on a common high speed backplane is what IBM refers to as "hybrid computing". It's a model that I think will be come a lot more common in the future. I think there are typically two types of hybrid models depending on whether the memory sharing is cache coherent or not. If you have coherent shared memory, the problem looks an awfully lot like virtualization. If you don't have coherent shared memory, then the shared memory basically becomes a pool to bounce into and out-of. PS - should I create a new thread on the two mailing lists mentioned above? I don't want to go too far off-topic in an alacrityvm thread. :) Couldn't hurt. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On Thu, Dec 24, 2009 at 11:09:39AM -0600, Anthony Liguori wrote: > On 12/23/2009 05:42 PM, Ira W. Snyder wrote: > > > > I've got a single PCI Host (master) with ~20 PCI slots. Physically, it > > is a backplane in a cPCI chassis, but the form factor is irrelevant. It > > is regular PCI from a software perspective. > > > > Into this backplane, I plug up to 20 PCI Agents (slaves). They are > > powerpc computers, almost identical to the Freescale MPC8349EMDS board. > > They're full-featured powerpc computers, with CPU, RAM, etc. They can > > run standalone. > > > > I want to use the PCI backplane as a data transport. Specifically, I > > want to transport ethernet over the backplane, so I can have the powerpc > > boards mount their rootfs via NFS, etc. Everyone knows how to write > > network daemons. It is a good and very well known way to transport data > > between systems. > > > > On the PCI bus, the powerpc systems expose 3 PCI BAR's. The size is > > configureable, as is the memory location at which they point. What I > > cannot do is get notified when a read/write hits the BAR. There is a > > feature on the board which allows me to generate interrupts in either > > direction: agent->master (PCI INTX) and master->agent (via an MMIO > > register). The PCI vendor ID and device ID are not configureable. > > > > One thing I cannot assume is that the PCI master system is capable of > > performing DMA. In my system, it is a Pentium3 class x86 machine, which > > has no DMA engine. However, the PowerPC systems do have DMA engines. In > > virtio terms, it was suggested to make the powerpc systems the "virtio > > hosts" (running the backends) and make the x86 (PCI master) the "virtio > > guest" (running virtio-net, etc.). > > IMHO, virtio and vbus are both the wrong model for what you're doing. > The key reason why is that virtio and vbus are generally designed around > the concept that there is shared cache coherent memory from which you > can use lock-less ring queues to implement efficient I/O. > > In your architecture, you do not have cache coherent shared memory. > Instead, you have two systems connected via a PCI backplace with > non-coherent shared memory. > > You probably need to use the shared memory as a bounce buffer and > implement a driver on top of that. > > > I'm not sure what you're suggesting in the paragraph above. I want to > > use virtio-net as the transport, I do not want to write my own > > virtual-network driver. Can you please clarify? > > virtio-net and vbus are going to be overly painful for you to use > because no one end can access arbitrary memory in the other end. > The PCI Agents (powerpc's) can access the lowest 4GB of the PCI Master's memory. Not all at the same time, but I have a 1GB movable window into PCI address space. I hunch Kyle's setup is similar. I've proved that virtio can work via my "crossed-wires" driver, hooking two virtio-net's together. With a proper in-kernel backend, I think the issues would be gone, and things would work great. > > Hopefully that explains what I'm trying to do. I'd love someone to help > > guide me in the right direction here. I want something to fill this need > > in mainline. > > If I were you, I would write a custom network driver. virtio-net is > awfully small (just a few hundred lines). I'd use that as a basis but I > would not tie into virtio or vbus. The paradigms don't match. > This is exactly what I did first. I proposed it for mainline, and David Miller shot it down, saying: you're creating your own virtualization scheme, use virtio instead. Arnd Bergmann is maintaining a driver out-of-tree for some IBM cell boards which is very similar, IIRC. In my driver, I used the PCI Agent's PCI BAR's to contain ring descriptors. The PCI Agent actually handles all data transfer (via the onboard DMA engine). It works great. I'll gladly post it if you'd like to see it. In my driver, I had to use 64K MTU to get acceptable performance. I'm not entirely sure how to implement a driver that can handle scatter/gather (fragmented skb's). It clearly isn't that easy to tune a network driver for good performance. For reference, my "crossed-wires" virtio drivers achieved excellent performance (10x better than my custom driver) with 1500 byte MTU. > > I've been contacted seperately by 10+ people also looking > > for a similar solution. I hunch most of them end up doing what I did: > > write a quick-and-dirty network driver. I've been working on this for a > > year, just to give an idea. > > The whole architecture of having multiple heterogenous systems on a > common high speed backplane is what IBM refers to as "hybrid computing". > It's a model that I think will be come a lot more common in the > future. I think there are typically two types of hybrid models > depending on whether the memory sharing is cache coherent or not. If > you have coherent shared memory, the problem looks an awfully lot like > virtualization. If you do
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/23/2009 11:21 PM, Gregory Haskins wrote: That said, you are still incorrect. With what I proposed, the model will run as an in-kernel vbus device, and no longer run in userspace. It would therefore improve virtio-net as I stated, much in the same way vhost-net or venet-tap do today. That can't work. virtio-net has its own ABI on top of virtio, for example it prepends a header for TSO information. Maybe if you disable all features it becomes compatible with venet, but that cripples it. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Alacrityvm-devel] [GIT PULL] AlacrityVM guest drivers for 2.6.33
On 12/27/09 4:15 AM, Avi Kivity wrote: > On 12/23/2009 11:21 PM, Gregory Haskins wrote: >> That said, you are still incorrect. With what I proposed, the model >> will run as an in-kernel vbus device, and no longer run in userspace. >> It would therefore improve virtio-net as I stated, much in the same >> way vhost-net or venet-tap do today. >> > > That can't work. virtio-net has its own ABI on top of virtio, for > example it prepends a header for TSO information. Maybe if you disable > all features it becomes compatible with venet, but that cripples it. > You are confused. The backend would be virtio-net specific, and would therefore understand the virtio-net ABI. It would support any feature of virtio-net as long as it was implemented and negotiated by both sides of the link. -Greg signature.asc Description: OpenPGP digital signature