Re: KVM for Sparc?
On 9/24/08, David Miller [EMAIL PROTECTED] wrote: From: Blue Swirl [EMAIL PROTECTED] Date: Wed, 24 Sep 2008 21:06:21 +0300 Now I found the relevant part in the manuals. The extra sun4v bit is not taken into account from user mode, so we can't catch privileged to hyperprivileged mode traps easily. That's right, the top bit is ignored in user mode. The hypervisor uses traps 0x80, 0x83, 0x84, 0x85, and 0xff. Looking at how these alias to low number traps: first four are unused or used for resets (SIR, RED, XIR), so they are not in the fast path. 0xff aliases to 0x7f, which is part of Fill 7 otherwin trap. Maybe that is not performance critical? The Fill 7 trap entry could be shortened with off-table jumps. I'm thinking that we could disassemble the calling instruction on entry to the lower traps and detect what was the true cause of the trap. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On 9/23/08, David Miller [EMAIL PROTECTED] wrote: From: Blue Swirl [EMAIL PROTECTED] Date: Tue, 23 Sep 2008 18:28:06 +0300 On 9/22/08, David Miller [EMAIL PROTECTED] wrote: As he mentioned, the V8 rett instruction causes problems on V9 chips. An opcode which was a V8 privileged instruction, rett, got reused as a non-privileged instruction in V9, for return. There are others: rdtbr/flushw and stdfq/stqf. Also any ASI 0x80 accesses are unprivileged on V9, though that shouldn't be a problem since all ASIs used on V8 were 0x80. And of course MMUs are incompatible. Thanks for the list. I sent a message to someone who I think might have been responsible for these architectual design decisions, letting them know what problems it is causing :-) So booting a 32-bit kernel on a 64-bit cpu is going to be challenging, at best. Maybe it would be possible to run V8 userspace with full speed acceleration on V9 and use translation only for kernel code? Yes, that should work. BTW, there is another area related the ASIs. Trap numbers. Even through V9, traps only up to 0x7f are valid. But sun4v extended V9 to allow trap numbers = 0x80, mostly these are used for hypervisor calls. The trap number field of the instruction is just extended one more bit higher to accomodate this. I see, also Qemu needs to use one more bit then. Does this mean that even V8 code written specially may use these traps to call hypervisor? Then we would need to catch these, maybe with the some assistance from the hypervisor. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On 9/23/08, David Miller [EMAIL PROTECTED] wrote: From: Blue Swirl [EMAIL PROTECTED] Date: Tue, 23 Sep 2008 18:34:12 +0300 On 9/23/08, David Miller [EMAIL PROTECTED] wrote: Sun4v systems come with Sun's hypervisor. Linux simply runs on top of that, whether as a host or a guest. The hypervisor source is opensource and we could technically make changes to it, but it isn't very practical. Do you mean OpenxVM? I think Sun has not published the hypervisor part yet. No, I mean OpenSPARC. The full hypervisor source code for sun4v is in the tarball. http://www.opensparc.net/ The code is there for both Niagara-T1 and Niagara-T2, for example for Niagara-T1 click on Get The Source -- OpenSPARC T1 -- Downloads Then on that page you want OpenSPARC T1 Download for Architecture and Performance Modeling Tools. It includes the full hypervisor and even the openboot PROM source code. I'm not sure, but I think that hypervisor is not a real hypervisor like the PROM version but one specially designed for Legion emulator. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On 9/24/08, Blue Swirl [EMAIL PROTECTED] wrote: On 9/23/08, David Miller [EMAIL PROTECTED] wrote: From: Blue Swirl [EMAIL PROTECTED] Date: Tue, 23 Sep 2008 18:28:06 +0300 On 9/22/08, David Miller [EMAIL PROTECTED] wrote: As he mentioned, the V8 rett instruction causes problems on V9 chips. An opcode which was a V8 privileged instruction, rett, got reused as a non-privileged instruction in V9, for return. There are others: rdtbr/flushw and stdfq/stqf. Also any ASI 0x80 accesses are unprivileged on V9, though that shouldn't be a problem since all ASIs used on V8 were 0x80. And of course MMUs are incompatible. Thanks for the list. I sent a message to someone who I think might have been responsible for these architectual design decisions, letting them know what problems it is causing :-) So booting a 32-bit kernel on a 64-bit cpu is going to be challenging, at best. Maybe it would be possible to run V8 userspace with full speed acceleration on V9 and use translation only for kernel code? Yes, that should work. BTW, there is another area related the ASIs. Trap numbers. Even through V9, traps only up to 0x7f are valid. But sun4v extended V9 to allow trap numbers = 0x80, mostly these are used for hypervisor calls. The trap number field of the instruction is just extended one more bit higher to accomodate this. I see, also Qemu needs to use one more bit then. Does this mean that even V8 code written specially may use these traps to call hypervisor? Then we would need to catch these, maybe with the some assistance from the hypervisor. Now I found the relevant part in the manuals. The extra sun4v bit is not taken into account from user mode, so we can't catch privileged to hyperprivileged mode traps easily. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Blue Swirl [EMAIL PROTECTED] Date: Wed, 24 Sep 2008 20:20:33 +0300 I see, also Qemu needs to use one more bit then. Does this mean that even V8 code written specially may use these traps to call hypervisor? No, V8 code should not set the extra bit. Only V9 code on a processor which is hypervisor capable should do this. And the hypervisor calls can only be invoked from privileged mode. This is all described in the UltraSPARC Architecture 2005 Specification (Hyperprivileged Edition) at: http://www.opensparc.net/opensparc-t1/index.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Blue Swirl [EMAIL PROTECTED] Date: Wed, 24 Sep 2008 20:22:45 +0300 I'm not sure, but I think that hypervisor is not a real hypervisor like the PROM version but one specially designed for Legion emulator. No, it's the real deal. All the real hardware device support is there. And Legion simulates a real, full Niagara system. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Blue Swirl [EMAIL PROTECTED] Date: Wed, 24 Sep 2008 21:06:21 +0300 Now I found the relevant part in the manuals. The extra sun4v bit is not taken into account from user mode, so we can't catch privileged to hyperprivileged mode traps easily. That's right, the top bit is ignored in user mode. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On 9/22/08, David Miller [EMAIL PROTECTED] wrote: From: Anthony Liguori [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 14:34:21 -0500 Blue Swirl wrote: But I think we could already start early drafting of what KVM support for Sparc32 and Sparc64 would mean. Because of certain problems in the V9 instruction set design (V8 rett reuse for example), it may be difficult or even impossible to use an accelerator if the host and target instruction sets do not match. I don't know much about the Sparc architecture, but the embedded PowerPC port that Hollis has spear-headed is for an architecture that does not natively support hardware virtualization. As long as Sparc meets all of the requirements to do this sort of virtualization (all privileged instructions are trappable when run in non-privileged mode), it should be rather straight forward. As he mentioned, the V8 rett instruction causes problems on V9 chips. An opcode which was a V8 privileged instruction, rett, got reused as a non-privileged instruction in V9, for return. There are others: rdtbr/flushw and stdfq/stqf. Also any ASI 0x80 accesses are unprivileged on V9, though that shouldn't be a problem since all ASIs used on V8 were 0x80. And of course MMUs are incompatible. So booting a 32-bit kernel on a 64-bit cpu is going to be challenging, at best. Maybe it would be possible to run V8 userspace with full speed acceleration on V9 and use translation only for kernel code? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On 9/23/08, David Miller [EMAIL PROTECTED] wrote: From: Hollis Blanchard [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 16:25:51 -0500 On Mon, 2008-09-22 at 13:26 -0700, David Miller wrote: From: Hollis Blanchard [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 15:18:39 -0500 It would be even more interesting to implement host support on the Sparc processors with hardware virtualization support. That's just a lot of protocol and userland work, rather than much kernel stuff. To me it's less interesting, to be honest. Not sure what you mean. On IBM POWER servers, the hardware's virtualization support is only accessible to an IBM-developed hypervisor that is packaged with the system. If you're saying Sparc is in the same position, I agree that's not very interesting. Sun4v systems come with Sun's hypervisor. Linux simply runs on top of that, whether as a host or a guest. The hypervisor source is opensource and we could technically make changes to it, but it isn't very practical. Do you mean OpenxVM? I think Sun has not published the hypervisor part yet. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On 9/23/08, David Miller [EMAIL PROTECTED] wrote: From: Hollis Blanchard [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 16:27:40 -0500 On Mon, 2008-09-22 at 15:31 -0500, Javier Guerra wrote: On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard [EMAIL PROTECTED] wrote: It would be even more interesting to implement host support on the Sparc processors with hardware virtualization support. Does Sparc processors also have 'virtualization support' as an extra feature? i thought that 'non virtualizationability' was an intel-specific limitation. (come on... creating a privileged mode and not trapping violations? who else would design like that?) Trapping on privileged instructions is enough to get you virtualization functionality, but for good performance you really want additional hardware support to avoid those traps in the first place. That's not really available on sparc. Looking at other intercept possibilities in AMD SVM, I'm not sure what would increase performance anyway except IO support. Because there are no dedicated IO instructions (except maybe some ASI accesses), the guest can be given direct HW access just by mapping the pages to guest address space (and adding IOMMU TLBs). -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On Mon, 2008-09-22 at 15:15 -0700, David Miller wrote: From: Hollis Blanchard [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 16:27:40 -0500 On Mon, 2008-09-22 at 15:31 -0500, Javier Guerra wrote: On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard [EMAIL PROTECTED] wrote: It would be even more interesting to implement host support on the Sparc processors with hardware virtualization support. Does Sparc processors also have 'virtualization support' as an extra feature? i thought that 'non virtualizationability' was an intel-specific limitation. (come on... creating a privileged mode and not trapping violations? who else would design like that?) Trapping on privileged instructions is enough to get you virtualization functionality, but for good performance you really want additional hardware support to avoid those traps in the first place. That's not really available on sparc. As far as I can see, you have a hypervisor privilege mode which allows your guests to execute privileged instructions like WRPR without trapping. Without that hardware support (with only 2 privilege levels), you'd be forced to run the guest kernel in user mode, trapping on every WRPR (for example). That's what I mean by hardware helping you to avoid traps. -- Hollis Blanchard IBM Linux Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Blue Swirl [EMAIL PROTECTED] Date: Tue, 23 Sep 2008 18:28:06 +0300 On 9/22/08, David Miller [EMAIL PROTECTED] wrote: As he mentioned, the V8 rett instruction causes problems on V9 chips. An opcode which was a V8 privileged instruction, rett, got reused as a non-privileged instruction in V9, for return. There are others: rdtbr/flushw and stdfq/stqf. Also any ASI 0x80 accesses are unprivileged on V9, though that shouldn't be a problem since all ASIs used on V8 were 0x80. And of course MMUs are incompatible. Thanks for the list. I sent a message to someone who I think might have been responsible for these architectual design decisions, letting them know what problems it is causing :-) So booting a 32-bit kernel on a 64-bit cpu is going to be challenging, at best. Maybe it would be possible to run V8 userspace with full speed acceleration on V9 and use translation only for kernel code? Yes, that should work. BTW, there is another area related the ASIs. Trap numbers. Even through V9, traps only up to 0x7f are valid. But sun4v extended V9 to allow trap numbers = 0x80, mostly these are used for hypervisor calls. The trap number field of the instruction is just extended one more bit higher to accomodate this. It turns out that binutils's assembler didn't range check the trap number field value, so binutils needed no changes to support this :-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Blue Swirl [EMAIL PROTECTED] Date: Tue, 23 Sep 2008 18:34:12 +0300 On 9/23/08, David Miller [EMAIL PROTECTED] wrote: Sun4v systems come with Sun's hypervisor. Linux simply runs on top of that, whether as a host or a guest. The hypervisor source is opensource and we could technically make changes to it, but it isn't very practical. Do you mean OpenxVM? I think Sun has not published the hypervisor part yet. No, I mean OpenSPARC. The full hypervisor source code for sun4v is in the tarball. http://www.opensparc.net/ The code is there for both Niagara-T1 and Niagara-T2, for example for Niagara-T1 click on Get The Source -- OpenSPARC T1 -- Downloads Then on that page you want OpenSPARC T1 Download for Architecture and Performance Modeling Tools. It includes the full hypervisor and even the openboot PROM source code. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Hollis Blanchard [EMAIL PROTECTED] Date: Tue, 23 Sep 2008 11:07:31 -0500 As far as I can see, you have a hypervisor privilege mode which allows your guests to execute privileged instructions like WRPR without trapping. Sure. Without that hardware support (with only 2 privilege levels), you'd be forced to run the guest kernel in user mode, trapping on every WRPR (for example). That's what I mean by hardware helping you to avoid traps. Understood. Actually, we'll likely need to interpret instructions or rewrite some of them when executing privileged mode code, because, as stated, several 32-bit V8 Sparc instructions which were privileged became non-privileged instructions for 64-bit V9. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
Blue Swirl wrote: Hi, Sorry for cross-posting (and because I used the wrong address in the first time for KVM, sorry for the duplicate on Sparclinux). Sparc host support for Qemu is getting close to ready, I can already run a Sparc32 system emulator on OpenBSD/Sparc64 and there is some limited success with recent glibc on Linux/Sparc64. Otherwise Sparc32 target emulator is pretty stable. Sparc64 target emulator can boot from several CD images, but crashes pretty soon. But I think we could already start early drafting of what KVM support for Sparc32 and Sparc64 would mean. Because of certain problems in the V9 instruction set design (V8 rett reuse for example), it may be difficult or even impossible to use an accelerator if the host and target instruction sets do not match. I don't know much about the Sparc architecture, but the embedded PowerPC port that Hollis has spear-headed is for an architecture that does not natively support hardware virtualization. As long as Sparc meets all of the requirements to do this sort of virtualization (all privileged instructions are trappable when run in non-privileged mode), it should be rather straight forward. KVM has code to do shadow paging and also software TLB virtualization, so depending on how Sparc manages the TLB, it should be pretty straight forward to base the Sparc code from the appropriate code that already exists. Other possibilities include porting kqemu or Xen, but I think KVM has the brightest future. I'm interested in pushing the Qemu side forward, but obviously something needs to be done by the kernel/KVM people too. KVM supports x86, ia64, s390, and PPC today. I don't think there would be any problems adding another architecture support. Almost all of the abstractions should have been flushed out already by the previous architecture ports. Moreover, since there is already good support for Sparc in QEMU, that should simplify things significantly. I'm looking forward to seeing the progress you make! Regards, Anthony Liguori What is the feeling on KVM and Sparclinux side? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
On Mon, 2008-09-22 at 14:34 -0500, Anthony Liguori wrote: Blue Swirl wrote: Hi, Sorry for cross-posting (and because I used the wrong address in the first time for KVM, sorry for the duplicate on Sparclinux). Sparc host support for Qemu is getting close to ready, I can already run a Sparc32 system emulator on OpenBSD/Sparc64 and there is some limited success with recent glibc on Linux/Sparc64. Otherwise Sparc32 target emulator is pretty stable. Sparc64 target emulator can boot from several CD images, but crashes pretty soon. But I think we could already start early drafting of what KVM support for Sparc32 and Sparc64 would mean. Because of certain problems in the V9 instruction set design (V8 rett reuse for example), it may be difficult or even impossible to use an accelerator if the host and target instruction sets do not match. I don't know much about the Sparc architecture, but the embedded PowerPC port that Hollis has spear-headed is for an architecture that does not natively support hardware virtualization. As long as Sparc meets all of the requirements to do this sort of virtualization (all privileged instructions are trappable when run in non-privileged mode), it should be rather straight forward. It would be even more interesting to implement host support on the Sparc processors with hardware virtualization support. Without that, functionality is possible (as we've demonstrated with KVM on PPC440)... the trick is performance. -- Hollis Blanchard IBM Linux Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Javier Guerra [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 15:31:07 -0500 On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard [EMAIL PROTECTED] wrote: It would be even more interesting to implement host support on the Sparc processors with hardware virtualization support. Does Sparc processors also have 'virtualization support' as an extra feature? Only sun4v processors (Niagara, etc.) have hypervisor support. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM for Sparc?
From: Hollis Blanchard [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 16:25:51 -0500 On Mon, 2008-09-22 at 13:26 -0700, David Miller wrote: From: Hollis Blanchard [EMAIL PROTECTED] Date: Mon, 22 Sep 2008 15:18:39 -0500 It would be even more interesting to implement host support on the Sparc processors with hardware virtualization support. That's just a lot of protocol and userland work, rather than much kernel stuff. To me it's less interesting, to be honest. Not sure what you mean. On IBM POWER servers, the hardware's virtualization support is only accessible to an IBM-developed hypervisor that is packaged with the system. If you're saying Sparc is in the same position, I agree that's not very interesting. Sun4v systems come with Sun's hypervisor. Linux simply runs on top of that, whether as a host or a guest. The hypervisor source is opensource and we could technically make changes to it, but it isn't very practical. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
KVM for Sparc?
Hi, Sorry for cross-posting (and because I used the wrong address in the first time for KVM, sorry for the duplicate on Sparclinux). Sparc host support for Qemu is getting close to ready, I can already run a Sparc32 system emulator on OpenBSD/Sparc64 and there is some limited success with recent glibc on Linux/Sparc64. Otherwise Sparc32 target emulator is pretty stable. Sparc64 target emulator can boot from several CD images, but crashes pretty soon. But I think we could already start early drafting of what KVM support for Sparc32 and Sparc64 would mean. Because of certain problems in the V9 instruction set design (V8 rett reuse for example), it may be difficult or even impossible to use an accelerator if the host and target instruction sets do not match. Other possibilities include porting kqemu or Xen, but I think KVM has the brightest future. I'm interested in pushing the Qemu side forward, but obviously something needs to be done by the kernel/KVM people too. What is the feeling on KVM and Sparclinux side? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
