From: Andy Lutomirski <[email protected]> All that paranoid exit needs to do is to disable IRQs, handle IRQ tracing, then restore CR3, and restore GS base. Simply do those actions in that order. Cleaning up the spaghetti code.
Signed-off-by: Andy Lutomirski <[email protected]> Signed-off-by: Chang S. Bae <[email protected]> Signed-off-by: Sasha Levin <[email protected]> Reviewed-by: Tony Luck <[email protected]> Cc: Thomas Gleixner <[email protected]> Cc: Borislav Petkov <[email protected]> Cc: Andy Lutomirski <[email protected]> Cc: H. Peter Anvin <[email protected]> Cc: Dave Hansen <[email protected]> Cc: Tony Luck <[email protected]> Cc: Andi Kleen <[email protected]> Cc: Vegard Nossum <[email protected]> --- arch/x86/entry/entry_64.S | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 3063aa9090f9a..0da56e6791b73 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1266,19 +1266,25 @@ SYM_CODE_END(paranoid_entry) SYM_CODE_START_LOCAL(paranoid_exit) UNWIND_HINT_REGS DISABLE_INTERRUPTS(CLBR_ANY) + + /* + * The order of operations is important. IRQ tracing requires + * kernel GS base and CR3. RESTORE_CR3 requires kernel GS base. + * + * NB to anyone to try to optimize this code: this code does + * not execute at all for exceptions from user mode. Those + * exceptions go through error_exit instead. + */ TRACE_IRQS_OFF_DEBUG - testl %ebx, %ebx /* swapgs needed? */ - jnz .Lparanoid_exit_no_swapgs - TRACE_IRQS_IRETQ - /* Always restore stashed CR3 value (see paranoid_entry) */ - RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 + RESTORE_CR3 scratch_reg=%rax save_reg=%r14 + + /* If EBX is 0, SWAPGS is required */ + testl %ebx, %ebx + jnz restore_regs_and_return_to_kernel + + /* We are returning to a context with user GS base */ SWAPGS_UNSAFE_STACK jmp restore_regs_and_return_to_kernel -.Lparanoid_exit_no_swapgs: - TRACE_IRQS_IRETQ_DEBUG - /* Always restore stashed CR3 value (see paranoid_entry) */ - RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 - jmp restore_regs_and_return_to_kernel SYM_CODE_END(paranoid_exit) /* -- 2.20.1
