Re: Additional compiler barrier required in sched_preempt_enable_no_resched?

[Vikram Mulukutla]

On 5/13/2016 7:58 AM, Peter Zijlstra wrote:
On Thu, May 12, 2016 at 11:39:47PM -0700, Vikram Mulukutla wrote:
Hi,
I came across a piece of engineering code that looked like:

preempt_disable();
/* --cut, lots of code-- */
preempt_enable_no_resched();
put_user()
preempt_disable();

(If you wish to seriously question the usage of the preempt API in this
manner, I unfortunately have no comment since I didn't write the code.)

I'm with Thomas here, that's broken and should not be done.

Ok. I did in fact zero in on this code by replacing each instance of 
preempt_enable_no_resched with preempt_enable one by one (there were 
several uses in the driver). I will ask the original developer to 
consider using preempt_enable.


This particular block of code was causing lockups and crashes on a certain
ARM64 device. The generated assembly revealed that the compiler was simply
optimizing out the increment and decrement of the preempt count, allowing
put_user to run without preemption enabled, causing all sorts of badness.
Since put_user doesn't actually access the preempt count and translates to
just a few instructions without any branching, I suppose that the compiler
figured it was OK to optimize.

The immediate solution is to add a compiler barrier to the code above, but
should sched_preempt_enable_no_resched have an additional compiler barrier
after (has one before already) the preempt-count decrement to prevent this
sort of thing?

I think the below would be sufficient; IIRC the compiler may not combine
or elide volatile operations.

---
  include/asm-generic/preempt.h | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/asm-generic/preempt.h b/include/asm-generic/preempt.h
index 5d8ffa3e6f8c..c1cde3577551 100644
--- a/include/asm-generic/preempt.h
+++ b/include/asm-generic/preempt.h
@@ -7,10 +7,10 @@

  static __always_inline int preempt_count(void)
  {
-       return current_thread_info()->preempt_count;
+       return READ_ONCE(current_thread_info()->preempt_count);
  }

-static __always_inline int *preempt_count_ptr(void)
+static __always_inline volatile int *preempt_count_ptr(void)
  {
        return &current_thread_info()->preempt_count;
  }


Thanks Peter, this patch worked for me. The compiler no longer optimizes 
out the increment/decrement of the preempt_count.

Thanks,
Vikram