Re: [PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
Hi Christoffer On 2017/10/17 3:59, Christoffer Dall wrote: > On Mon, Oct 16, 2017 at 04:10:01PM +, gengdongjiu wrote: >> Hi Marc, >> >>> >>> Please also update the 32bit code accordingly, as it looks broken too. >> >> I have updated the 32 bit code according, in my hand, there is no arm32 host >> environment, >> So there is no method to verify it in the arm32 host, only verify the patch >> in the arm64 host. >> >> Anyway I firstly send the patch out for review. Thanks. >> > In this case, if you just clearly specify in the patches you send out > that the 32-bit one is untested, you can ask someone to test it for you. > Thanks for your reminder, today I found a arm32 board and have tested it. So I have tested in both arm32 and arm64. please review it. thanks. > Thanks, > -Christoffer > > . > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
On Mon, Oct 16, 2017 at 04:10:01PM +, gengdongjiu wrote: > Hi Marc, > > > > > Please also update the 32bit code accordingly, as it looks broken too. > > I have updated the 32 bit code according, in my hand, there is no arm32 host > environment, > So there is no method to verify it in the arm32 host, only verify the patch > in the arm64 host. > > Anyway I firstly send the patch out for review. Thanks. > In this case, if you just clearly specify in the patches you send out that the 32-bit one is untested, you can ask someone to test it for you. Thanks, -Christoffer ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
Hi Marc, > > Please also update the 32bit code accordingly, as it looks broken too. I have updated the 32 bit code according, in my hand, there is no arm32 host environment, So there is no method to verify it in the arm32 host, only verify the patch in the arm64 host. Anyway I firstly send the patch out for review. Thanks. > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
Hi Marc, On 2017/10/13 23:12, Marc Zyngier wrote: > On 13/10/17 15:29, gengdongjiu wrote: >> Hi Marc, >> Thank you very much for your time to review it. >> >>> On 12/10/17 17:44, Dongjiu Geng wrote: When a exception is trapped to EL2, hardware uses ELR_ELx to hold the current fault instruction address. If KVM wants to inject a abort to 32 bit guest, it needs to set the LR register for the guest to emulate this abort happened in the guest. Because ARM32 architecture is Multi-pipeline, so the LR value has an offset to >>> >>> What does "Multi-pipeline" mean? >> >> I mean the ARM's single-cycle instruction 3-stage pipeline operation, as >> shown below: >> >> fetch decode execute >> fetchdecode execute >> fetchdecode execute >> >> when CPU finish instructions fetch, PC=PC + 4 >> when CPU finish instructions decode, PC=PC + 8 >> when CPU finish instructions execution, PC=PC+12 > > Yeah, and that's called pipelined execution. "Multi-pipeline" doesn't > mean anything. Also, that's an artefact of the original ARM1 > implementation, and not how modern CPUs work anymore. Ok, thanks for the clarification. > >> >> that is to say, when happen data abort, >> the PC = fault instruction address + 12, LR_abt = fault instruction address >> + 8 >> >> In order to emulate this abort for KVM, LR_abt needs to add an offset 8 when >> inject data abort. >> >>> the fault instruction address. The offsets applied to Link value for exceptions as shown below, which should be added for the ARM32 link register(LR). Exception Offset, for PE state of: A32 T32 Undefined Instruction +4+2 Prefetch Abort +4+4 Data Abort +8+8 IRQ or FIQ +4+4 >>> >>> Please document where this table is coming from. >> >> >> Thanks for pointing out. Will add it. >> It come from: DDI0487A_k_armv8_arm_iss10775, "G1.12.3 Overview of exception >> entry", Table G1-10 Offsets applied to Link value for exceptions taken to >> non-EL2 modes >> >>> Signed-off-by: Dongjiu GengSigned-off-by: Haibin Zhang --- For example, to the undefined instruction injection: 1. Guest OS call SMC(Secure Monitor Call) instruction in the address 0xc025405c, then Guest traps to hypervisor c0254050: e59d5028ldr r5, [sp, #40] ; 0x28 c0254054: e3a03001mov r3, #1 c0254058: e1a01003mov r1, r3 c025405c: e1600070smc 0 c0254060: e30a0270movwr0, #41584 ; 0xa270 c0254064: e34c00bfmovtr0, #49343 ; 0xc0bf 2. KVM injects undefined abort to guest 3. We will find the fault PC is 0xc0254058, not 0xc025405c. [ 12.348072] Internal error: Oops - undefined instruction: 0 [#1] SMP ARM [ 12.349786] Modules linked in: [ 12.350563] CPU: 1 PID: 71 Comm: cat Not tainted 4.1.0-dirty #25 [ 12.352061] Hardware name: Generic DT based system [ 12.353275] task: d9d08000 ti: d9cfe000 task.ti: d9cfe000 [ 12.354637] PC is at proc_dointvec+0x20/0x60 [ 12.355717] LR is at proc_sys_call_handler+0xb0/0xc4 [ 12.356972] pc : []lr : []psr: a0060013 [ 12.356972] sp : d9cffe90 ip : c0254038 fp : 0001 [ 12.359824] r10: d9cfff80 r9 : 0004 r8 : [ 12.361132] r7 : bec21cb0 r6 : d9cffec4 r5 : d9cfff80 r4 : c0e82de0 [ 12.362766] r3 : 0001 r2 : bec21cb0 r1 : 0001 r0 : c0e82de0 [ 12.364400] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [ 12.366183] Control: 10c5383d Table: 59d3406a DAC: 0015 [ 12.367623] Process cat (pid: 71, stack limit = 0xd9cfe220) 4. After correct the LR register, it will have right value [ 125.763370] Internal error: Oops - undefined instruction: 0 [#2] SMP ARM [ 125.767010] Modules linked in: [ 125.768472] CPU: 1 PID: 74 Comm: cat Tainted: G D 4.1.0-dirty #25 [ 125.771854] Hardware name: Generic DT based system [ 125.774053] task: db0bb900 ti: d9d1 task.ti: d9d1 [ 125.776821] PC is at proc_dointvec+0x24/0x60 [ 125.778919] LR is at proc_sys_call_handler+0xb0/0xc4 [ 125.781269] pc : []lr : []psr: a0060013 [ 125.781269] sp : d9d11e90 ip : c0254038 fp : 0001 [ 125.786581] r10: d9d11f80 r9 : 0004 r8 : [ 125.789673] r7 : be92ccb0 r6 : d9d11ec4 r5 : d9d11f80 r4 : c0e82de0 [ 125.792828] r3 : 0001 r2 : be92ccb0 r1 : 0001 r0 : c0e82de0 [ 125.795890] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user For other exception injection, such as Data/Prefetch abort, also needs
Re: [PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
On 13/10/17 15:29, gengdongjiu wrote: > Hi Marc, > Thank you very much for your time to review it. > >> On 12/10/17 17:44, Dongjiu Geng wrote: >>> When a exception is trapped to EL2, hardware uses ELR_ELx to hold the >>> current fault instruction address. If KVM wants to inject a abort to >>> 32 bit guest, it needs to set the LR register for the guest to emulate >>> this abort happened in the guest. Because ARM32 architecture is >>> Multi-pipeline, so the LR value has an offset to >> >> What does "Multi-pipeline" mean? > > I mean the ARM's single-cycle instruction 3-stage pipeline operation, as > shown below: > > fetch decode execute > fetchdecode execute > fetchdecode execute > > when CPU finish instructions fetch, PC=PC + 4 > when CPU finish instructions decode, PC=PC + 8 > when CPU finish instructions execution, PC=PC+12 Yeah, and that's called pipelined execution. "Multi-pipeline" doesn't mean anything. Also, that's an artefact of the original ARM1 implementation, and not how modern CPUs work anymore. > > that is to say, when happen data abort, > the PC = fault instruction address + 12, LR_abt = fault instruction address + > 8 > > In order to emulate this abort for KVM, LR_abt needs to add an offset 8 when > inject data abort. > >> >>> the fault instruction address. >>> >>> The offsets applied to Link value for exceptions as shown below, which >>> should be added for the ARM32 link register(LR). >>> >>> Exception Offset, for PE state of: >>> A32 T32 >>> Undefined Instruction +4+2 >>> Prefetch Abort +4+4 >>> Data Abort +8+8 >>> IRQ or FIQ +4+4 >> >> Please document where this table is coming from. > > > Thanks for pointing out. Will add it. > It come from: DDI0487A_k_armv8_arm_iss10775, "G1.12.3 Overview of exception > entry", Table G1-10 Offsets applied to Link value for exceptions taken to > non-EL2 modes > >> >>> >>> Signed-off-by: Dongjiu Geng>>> Signed-off-by: Haibin Zhang >>> >>> --- >>> For example, to the undefined instruction injection: >>> >>> 1. Guest OS call SMC(Secure Monitor Call) instruction in the address >>> 0xc025405c, then Guest traps to hypervisor >>> >>> c0254050: e59d5028ldr r5, [sp, #40] ; 0x28 >>> c0254054: e3a03001mov r3, #1 >>> c0254058: e1a01003mov r1, r3 >>> c025405c: e1600070smc 0 >>> c0254060: e30a0270movwr0, #41584 ; 0xa270 >>> c0254064: e34c00bfmovtr0, #49343 ; 0xc0bf >>> >>> 2. KVM injects undefined abort to guest 3. We will find the fault PC >>> is 0xc0254058, not 0xc025405c. >>> >>> [ 12.348072] Internal error: Oops - undefined instruction: 0 [#1] SMP ARM >>> [ 12.349786] Modules linked in: >>> [ 12.350563] CPU: 1 PID: 71 Comm: cat Not tainted 4.1.0-dirty #25 >>> [ 12.352061] Hardware name: Generic DT based system >>> [ 12.353275] task: d9d08000 ti: d9cfe000 task.ti: d9cfe000 >>> [ 12.354637] PC is at proc_dointvec+0x20/0x60 >>> [ 12.355717] LR is at proc_sys_call_handler+0xb0/0xc4 >>> [ 12.356972] pc : []lr : []psr: a0060013 >>> [ 12.356972] sp : d9cffe90 ip : c0254038 fp : 0001 >>> [ 12.359824] r10: d9cfff80 r9 : 0004 r8 : >>> [ 12.361132] r7 : bec21cb0 r6 : d9cffec4 r5 : d9cfff80 r4 : c0e82de0 >>> [ 12.362766] r3 : 0001 r2 : bec21cb0 r1 : 0001 r0 : c0e82de0 >>> [ 12.364400] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment >>> user >>> [ 12.366183] Control: 10c5383d Table: 59d3406a DAC: 0015 >>> [ 12.367623] Process cat (pid: 71, stack limit = 0xd9cfe220) >>> >>> 4. After correct the LR register, it will have right value >>> >>> [ 125.763370] Internal error: Oops - undefined instruction: 0 [#2] >>> SMP ARM [ 125.767010] Modules linked in: >>> [ 125.768472] CPU: 1 PID: 74 Comm: cat Tainted: G D >>> 4.1.0-dirty #25 >>> [ 125.771854] Hardware name: Generic DT based system [ 125.774053] >>> task: db0bb900 ti: d9d1 task.ti: d9d1 [ 125.776821] PC is at >>> proc_dointvec+0x24/0x60 [ 125.778919] LR is at >>> proc_sys_call_handler+0xb0/0xc4 >>> [ 125.781269] pc : []lr : []psr: a0060013 >>> [ 125.781269] sp : d9d11e90 ip : c0254038 fp : 0001 [ >>> 125.786581] r10: d9d11f80 r9 : 0004 r8 : [ 125.789673] >>> r7 : be92ccb0 r6 : d9d11ec4 r5 : d9d11f80 r4 : c0e82de0 [ >>> 125.792828] r3 : 0001 r2 : be92ccb0 r1 : 0001 r0 : c0e82de0 >>> [ 125.795890] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM >>> Segment user >>> >>> For other exception injection, such as Data/Prefetch abort, also needs >>> to correct >>> --- >>> arch/arm64/kvm/inject_fault.c | 18 -- >>> 1 file changed, 12 insertions(+), 6 deletions(-) >>> >>> diff --git a/arch/arm64/kvm/inject_fault.c >>>
Re: [PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
Hi Marc, Thank you very much for your time to review it. > On 12/10/17 17:44, Dongjiu Geng wrote: > > When a exception is trapped to EL2, hardware uses ELR_ELx to hold the > > current fault instruction address. If KVM wants to inject a abort to > > 32 bit guest, it needs to set the LR register for the guest to emulate > > this abort happened in the guest. Because ARM32 architecture is > > Multi-pipeline, so the LR value has an offset to > > What does "Multi-pipeline" mean? I mean the ARM's single-cycle instruction 3-stage pipeline operation, as shown below: fetch decode execute fetchdecode execute fetchdecode execute when CPU finish instructions fetch, PC=PC + 4 when CPU finish instructions decode, PC=PC + 8 when CPU finish instructions execution, PC=PC+12 that is to say, when happen data abort, the PC = fault instruction address + 12, LR_abt = fault instruction address + 8 In order to emulate this abort for KVM, LR_abt needs to add an offset 8 when inject data abort. > > > the fault instruction address. > > > > The offsets applied to Link value for exceptions as shown below, which > > should be added for the ARM32 link register(LR). > > > > Exception Offset, for PE state of: > > A32 T32 > > Undefined Instruction +4+2 > > Prefetch Abort +4+4 > > Data Abort +8+8 > > IRQ or FIQ +4+4 > > Please document where this table is coming from. Thanks for pointing out. Will add it. It come from: DDI0487A_k_armv8_arm_iss10775, "G1.12.3 Overview of exception entry", Table G1-10 Offsets applied to Link value for exceptions taken to non-EL2 modes > > > > > Signed-off-by: Dongjiu Geng> > Signed-off-by: Haibin Zhang > > > > --- > > For example, to the undefined instruction injection: > > > > 1. Guest OS call SMC(Secure Monitor Call) instruction in the address > > 0xc025405c, then Guest traps to hypervisor > > > > c0254050: e59d5028ldr r5, [sp, #40] ; 0x28 > > c0254054: e3a03001mov r3, #1 > > c0254058: e1a01003mov r1, r3 > > c025405c: e1600070smc 0 > > c0254060: e30a0270movwr0, #41584 ; 0xa270 > > c0254064: e34c00bfmovtr0, #49343 ; 0xc0bf > > > > 2. KVM injects undefined abort to guest 3. We will find the fault PC > > is 0xc0254058, not 0xc025405c. > > > > [ 12.348072] Internal error: Oops - undefined instruction: 0 [#1] SMP ARM > > [ 12.349786] Modules linked in: > > [ 12.350563] CPU: 1 PID: 71 Comm: cat Not tainted 4.1.0-dirty #25 > > [ 12.352061] Hardware name: Generic DT based system > > [ 12.353275] task: d9d08000 ti: d9cfe000 task.ti: d9cfe000 > > [ 12.354637] PC is at proc_dointvec+0x20/0x60 > > [ 12.355717] LR is at proc_sys_call_handler+0xb0/0xc4 > > [ 12.356972] pc : []lr : []psr: a0060013 > > [ 12.356972] sp : d9cffe90 ip : c0254038 fp : 0001 > > [ 12.359824] r10: d9cfff80 r9 : 0004 r8 : > > [ 12.361132] r7 : bec21cb0 r6 : d9cffec4 r5 : d9cfff80 r4 : c0e82de0 > > [ 12.362766] r3 : 0001 r2 : bec21cb0 r1 : 0001 r0 : c0e82de0 > > [ 12.364400] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment > > user > > [ 12.366183] Control: 10c5383d Table: 59d3406a DAC: 0015 > > [ 12.367623] Process cat (pid: 71, stack limit = 0xd9cfe220) > > > > 4. After correct the LR register, it will have right value > > > > [ 125.763370] Internal error: Oops - undefined instruction: 0 [#2] > > SMP ARM [ 125.767010] Modules linked in: > > [ 125.768472] CPU: 1 PID: 74 Comm: cat Tainted: G D > > 4.1.0-dirty #25 > > [ 125.771854] Hardware name: Generic DT based system [ 125.774053] > > task: db0bb900 ti: d9d1 task.ti: d9d1 [ 125.776821] PC is at > > proc_dointvec+0x24/0x60 [ 125.778919] LR is at > > proc_sys_call_handler+0xb0/0xc4 > > [ 125.781269] pc : []lr : []psr: a0060013 > > [ 125.781269] sp : d9d11e90 ip : c0254038 fp : 0001 [ > > 125.786581] r10: d9d11f80 r9 : 0004 r8 : [ 125.789673] > > r7 : be92ccb0 r6 : d9d11ec4 r5 : d9d11f80 r4 : c0e82de0 [ > > 125.792828] r3 : 0001 r2 : be92ccb0 r1 : 0001 r0 : c0e82de0 > > [ 125.795890] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM > > Segment user > > > > For other exception injection, such as Data/Prefetch abort, also needs > > to correct > > --- > > arch/arm64/kvm/inject_fault.c | 18 -- > > 1 file changed, 12 insertions(+), 6 deletions(-) > > > > diff --git a/arch/arm64/kvm/inject_fault.c > > b/arch/arm64/kvm/inject_fault.c index da6a8cf..da93508 100644 > > --- a/arch/arm64/kvm/inject_fault.c > > +++ b/arch/arm64/kvm/inject_fault.c > > @@ -33,12 +33,11 @@ > > #define LOWER_EL_AArch64_VECTOR0x400 > > #define LOWER_EL_AArch32_VECTOR0x600 > > > > -static void
[PATCH] arm64: KVM: set right LR register value for 32 bit guest when inject abort
When a exception is trapped to EL2, hardware uses ELR_ELx to hold the current fault instruction address. If KVM wants to inject a abort to 32 bit guest, it needs to set the LR register for the guest to emulate this abort happened in the guest. Because ARM32 architecture is Multi-pipeline, so the LR value has an offset to the fault instruction address. The offsets applied to Link value for exceptions as shown below, which should be added for the ARM32 link register(LR). Exception Offset, for PE state of: A32 T32 Undefined Instruction +4+2 Prefetch Abort +4+4 Data Abort +8+8 IRQ or FIQ +4+4 Signed-off-by: Dongjiu GengSigned-off-by: Haibin Zhang --- For example, to the undefined instruction injection: 1. Guest OS call SMC(Secure Monitor Call) instruction in the address 0xc025405c, then Guest traps to hypervisor c0254050: e59d5028ldr r5, [sp, #40] ; 0x28 c0254054: e3a03001mov r3, #1 c0254058: e1a01003mov r1, r3 c025405c: e1600070smc 0 c0254060: e30a0270movwr0, #41584 ; 0xa270 c0254064: e34c00bfmovtr0, #49343 ; 0xc0bf 2. KVM injects undefined abort to guest 3. We will find the fault PC is 0xc0254058, not 0xc025405c. [ 12.348072] Internal error: Oops - undefined instruction: 0 [#1] SMP ARM [ 12.349786] Modules linked in: [ 12.350563] CPU: 1 PID: 71 Comm: cat Not tainted 4.1.0-dirty #25 [ 12.352061] Hardware name: Generic DT based system [ 12.353275] task: d9d08000 ti: d9cfe000 task.ti: d9cfe000 [ 12.354637] PC is at proc_dointvec+0x20/0x60 [ 12.355717] LR is at proc_sys_call_handler+0xb0/0xc4 [ 12.356972] pc : []lr : []psr: a0060013 [ 12.356972] sp : d9cffe90 ip : c0254038 fp : 0001 [ 12.359824] r10: d9cfff80 r9 : 0004 r8 : [ 12.361132] r7 : bec21cb0 r6 : d9cffec4 r5 : d9cfff80 r4 : c0e82de0 [ 12.362766] r3 : 0001 r2 : bec21cb0 r1 : 0001 r0 : c0e82de0 [ 12.364400] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [ 12.366183] Control: 10c5383d Table: 59d3406a DAC: 0015 [ 12.367623] Process cat (pid: 71, stack limit = 0xd9cfe220) 4. After correct the LR register, it will have right value [ 125.763370] Internal error: Oops - undefined instruction: 0 [#2] SMP ARM [ 125.767010] Modules linked in: [ 125.768472] CPU: 1 PID: 74 Comm: cat Tainted: G D 4.1.0-dirty #25 [ 125.771854] Hardware name: Generic DT based system [ 125.774053] task: db0bb900 ti: d9d1 task.ti: d9d1 [ 125.776821] PC is at proc_dointvec+0x24/0x60 [ 125.778919] LR is at proc_sys_call_handler+0xb0/0xc4 [ 125.781269] pc : []lr : []psr: a0060013 [ 125.781269] sp : d9d11e90 ip : c0254038 fp : 0001 [ 125.786581] r10: d9d11f80 r9 : 0004 r8 : [ 125.789673] r7 : be92ccb0 r6 : d9d11ec4 r5 : d9d11f80 r4 : c0e82de0 [ 125.792828] r3 : 0001 r2 : be92ccb0 r1 : 0001 r0 : c0e82de0 [ 125.795890] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user For other exception injection, such as Data/Prefetch abort, also needs to correct --- arch/arm64/kvm/inject_fault.c | 18 -- 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c index da6a8cf..da93508 100644 --- a/arch/arm64/kvm/inject_fault.c +++ b/arch/arm64/kvm/inject_fault.c @@ -33,12 +33,11 @@ #define LOWER_EL_AArch64_VECTOR0x400 #define LOWER_EL_AArch32_VECTOR0x600 -static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset) +static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset, + u32 return_offset) { unsigned long cpsr; unsigned long new_spsr_value = *vcpu_cpsr(vcpu); - bool is_thumb = (new_spsr_value & COMPAT_PSR_T_BIT); - u32 return_offset = (is_thumb) ? 4 : 0; u32 sctlr = vcpu_cp15(vcpu, c1_SCTLR); cpsr = mode | COMPAT_PSR_I_BIT; @@ -65,7 +64,11 @@ static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset) static void inject_undef32(struct kvm_vcpu *vcpu) { - prepare_fault32(vcpu, COMPAT_PSR_MODE_UND, 4); + unsigned long spsr_value = *vcpu_cpsr(vcpu); + bool is_thumb = (spsr_value & COMPAT_PSR_T_BIT); + u32 return_offset = (is_thumb) ? 2 : 4; + + prepare_fault32(vcpu, COMPAT_PSR_MODE_UND, 4, return_offset); } /* @@ -75,21 +78,24 @@ static void inject_undef32(struct kvm_vcpu *vcpu) static void inject_abt32(struct kvm_vcpu *vcpu, bool is_pabt, unsigned long addr) { - u32 vect_offset; + u32 vect_offset, return_offset; u32 *far, *fsr; bool is_lpae; if (is_pabt) { vect_offset = 12; +