From: Colin Ian King <[email protected]> Date: Mon, 7 Jul 2025 14:23:40 +0100
> Analysis with gcov while running the stress-ng urandom stressor > shows that there are a couple of fortify panic paths that are highly > unlikely to be executed for well-behaving code. Adding appropriate > branch hints improves the stress-ng urandom stressor my a small > but statistically measureable amount. Ran 100 x 1 minute tests and > measured the stressor bogo-op rates on a Debian based Intel(R) > Core(TM) Ultra 9 285K with a 6.15 kernel with turbo disabled to > reduce jitter. > > Results based on a Geometic Mean of 100 tests: > > Without patch: 50512.95 bogo-ops/sec > With patch: 50819.58 bogo-ops/sec > > %Std.Deviation of ~0.18%, so low jitter in results, improvement of ~0.6% > > Branch hints can only be enabled if CONFIG_TRACE_BRANCH_PROFILING due > to a static variable being declared in __branch_check__ when using > trace branch profiling in the unlikely macro causing build issues. > > Signed-off-by: Colin Ian King <[email protected]> > --- > V2: add fortify_unlikely macro wrapper for CONFIG_TRACE_BRANCH_PROFILING > build config. > --- > include/linux/fortify-string.h | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h > index e4ce1cae03bf..c740114bcbf8 100644 > --- a/include/linux/fortify-string.h > +++ b/include/linux/fortify-string.h > @@ -10,6 +10,12 @@ > #define __FORTIFY_INLINE extern __always_inline __gnu_inline __overloadable > #define __RENAME(x) __asm__(#x) > > +#if defined(CONFIG_TRACE_BRANCH_PROFILING) > +#define fortify_unlikely(x) (x) > +#else > +#define fortify_unlikely(x) unlikely(x) > +#endif > + > #define FORTIFY_REASON_DIR(r) FIELD_GET(BIT(0), r) > #define FORTIFY_REASON_FUNC(r) FIELD_GET(GENMASK(7, 1), r) > #define FORTIFY_REASON(func, write) (FIELD_PREP(BIT(0), write) | \ > @@ -593,9 +599,9 @@ __FORTIFY_INLINE bool fortify_memcpy_chk(__kernel_size_t > size, Why only memcpy()? > * (The SIZE_MAX test is to optimize away checks where the buffer > * lengths are unknown.) > */ > - if (p_size != SIZE_MAX && p_size < size) > + if (fortify_unlikely(p_size != SIZE_MAX && p_size < size)) > fortify_panic(func, FORTIFY_WRITE, p_size, size, true); > - else if (q_size != SIZE_MAX && q_size < size) > + else if (fortify_unlikely(q_size != SIZE_MAX && q_size < size)) > fortify_panic(func, FORTIFY_READ, p_size, size, true); > > /* Thanks, Olek
