Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
+++ Stephen Boyd [17/04/21 18:52 -0700]: [snip] Sounds good. It means that some more ifdefs are probably required vs. making the array size be 0 when the config is disabled but that isn't a big problem for me. I'm reworking the code now and will test and then send v5 shortly. Thanks! Great, thanks a lot Stephen! Jessica
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
Quoting Jessica Yu (2021-04-15 06:04:35)
> +++ Stephen Boyd [09/04/21 18:52 -0700]:
> >diff --git a/include/linux/module.h b/include/linux/module.h
> >index 59f094fa6f74..4bf869f6c944 100644
> >--- a/include/linux/module.h
> >+++ b/include/linux/module.h
> >@@ -11,6 +11,7 @@
> >
> > #include
> > #include
> >+#include
> > #include
> > #include
> > #include
> >@@ -367,6 +368,9 @@ struct module {
> > /* Unique handle for this module */
> > char name[MODULE_NAME_LEN];
> >
> >+ /* Module build ID */
> >+ unsigned char build_id[BUILD_ID_SIZE_MAX];
>
> Hi Stephen,
>
> Since this field is not used when !CONFIG_STACKTRACE_BUILD_ID, I
> would prefer to wrap this in an ifdef, similar to the other
> CONFIG-dependent fields in struct module. This makes it explicit under
> what conditions (i.e. config) the field is meant to be used.
Ok will do.
> >diff --git a/kernel/module.c b/kernel/module.c
> >index 30479355ab85..6f5bc1b046a5 100644
> >--- a/kernel/module.c
> >+++ b/kernel/module.c
> >@@ -2770,6 +2771,20 @@ static void add_kallsyms(struct module *mod, const
> >struct load_info *info)
> > }
> > mod->core_kallsyms.num_symtab = ndst;
> > }
> >+
> >+static void init_build_id(struct module *mod, const struct load_info *info)
> >+{
> >+ const Elf_Shdr *sechdr;
> >+ unsigned int i;
> >+
> >+ for (i = 0; i < info->hdr->e_shnum; i++) {
> >+ sechdr = &info->sechdrs[i];
> >+ if (!sect_empty(sechdr) && sechdr->sh_type == SHT_NOTE &&
> >+ !build_id_parse_buf((void *)sechdr->sh_addr,
> >mod->build_id,
> >+ sechdr->sh_size))
> >+ break;
> >+ }
>
> If mod->build_id is not used when !CONFIG_STACKTRACE_BUILD_ID, then we
> don't need to look for it. I would be fine with wrapping the function
> body in an ifdef (similar to what we currently do in
> del_usage_links() and do_mod_ctors()).
Ok, done.
>
> >+}
> > #else
> > static inline void layout_symtab(struct module *mod, struct load_info *info)
> > {
> >@@ -2778,6 +2793,10 @@ static inline void layout_symtab(struct module *mod,
> >struct load_info *info)
> > static void add_kallsyms(struct module *mod, const struct load_info *info)
> > {
> > }
> >+
> >+static void init_build_id(struct module *mod, const struct load_info *info)
> >+{
> >+}
> > #endif /* CONFIG_KALLSYMS */
> >
> > static void dynamic_debug_setup(struct module *mod, struct _ddebug *debug,
> > unsigned int num)
> >@@ -4004,6 +4023,7 @@ static int load_module(struct load_info *info, const
> >char __user *uargs,
> > goto free_arch_cleanup;
> > }
> >
> >+ init_build_id(mod, info);
> > dynamic_debug_setup(mod, info->debug, info->num_debug);
> >
> > /* Ftrace init must be called in the MODULE_STATE_UNFORMED state */
> >@@ -4235,7 +4255,7 @@ void * __weak
> >dereference_module_function_descriptor(struct module *mod,
> > const char *module_address_lookup(unsigned long addr,
> > unsigned long *size,
> > unsigned long *offset,
> >- char **modname,
> >+ char **modname, const unsigned char **modbuildid,
> > char *namebuf)
> > {
> > const char *ret = NULL;
> >@@ -4246,6 +4266,8 @@ const char *module_address_lookup(unsigned long addr,
> > if (mod) {
> > if (modname)
> > *modname = mod->name;
> >+ if (modbuildid)
> >+ *modbuildid = mod->build_id;
>
> Then maybe we can set *modbuildid = NULL in the case of
> !CONFIG_STACKTRACE_BUILD_ID, similar to the kernel case in
> kallsyms_lookup_buildid().
>
Sounds good. It means that some more ifdefs are probably required vs.
making the array size be 0 when the config is disabled but that isn't a
big problem for me. I'm reworking the code now and will test and then
send v5 shortly. Thanks!
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
+++ Stephen Boyd [09/04/21 18:52 -0700]: Let's make kernel stacktraces easier to identify by including the build ID[1] of a module if the stacktrace is printing a symbol from a module. This makes it simpler for developers to locate a kernel module's full debuginfo for a particular stacktrace. Combined with scripts/decode_stracktrace.sh, a developer can download the matching debuginfo from a debuginfod[2] server and find the exact file and line number for the functions plus offsets in a stacktrace that match the module. This is especially useful for pstore crash debugging where the kernel crashes are recorded in something like console-ramoops and the recovery kernel/modules are different or the debuginfo doesn't exist on the device due to space concerns (the debuginfo can be too large for space limited devices). Originally, I put this on the %pS format, but that was quickly rejected given that %pS is used in other places such as ftrace where build IDs aren't meaningful. There was some discussions on the list to put every module build ID into the "Modules linked in:" section of the stacktrace message but that quickly becomes very hard to read once you have more than three or four modules linked in. It also provides too much information when we don't expect each module to be traversed in a stacktrace. Having the build ID for modules that aren't important just makes things messy. Splitting it to multiple lines for each module quickly explodes the number of lines printed in an oops too, possibly wrapping the warning off the console. And finally, trying to stash away each module used in a callstack to provide the ID of each symbol printed is cumbersome and would require changes to each architecture to stash away modules and return their build IDs once unwinding has completed. Instead, we opt for the simpler approach of introducing new printk formats '%pS[R]b' for "pointer symbolic backtrace with module build ID" and '%pBb' for "pointer backtrace with module build ID" and then updating the few places in the architecture layer where the stacktrace is printed to use this new format. Example: WARNING: CPU: 3 PID: 3373 at drivers/misc/lkdtm/bugs.c:83 lkdtm_WARNING+0x28/0x30 [lkdtm] Modules linked in: lkdtm rfcomm algif_hash algif_skcipher af_alg xt_cgroup uinput xt_MASQUERADE hci_uart CPU: 3 PID: 3373 Comm: bash Not tainted 5.11 #12 a8c0d47f7051f3e6670ceaea724af66a39c6cec8 Hardware name: Google Lazor (rev3+) with KB Backlight (DT) pstate: 0049 (nzcv daif +PAN -UAO -TCO BTYPE=--) pc : lkdtm_WARNING+0x28/0x30 [lkdtm] lr : lkdtm_do_action+0x24/0x40 [lkdtm] sp : ffc013febca0 x29: ffc013febca0 x28: ff88d9438040 x27: x26: x25: x24: ffdd0e9772c0 x23: 0020 x22: ffdd0e975366 x21: ffdd0e9772e0 x20: ffc013febde0 x19: 0008 x18: x17: x16: 0037 x15: ffdd102ab174 x14: 0003 x13: 0004 x12: x11: x10: x9 : 0001 x8 : ffdd0e979000 x7 : x6 : ffdd10ff6b54 x5 : x4 : x3 : ffc013feb938 x2 : ff89fef05a70 x1 : ff89feef5788 x0 : ffdd0e9772e0 Call trace: lkdtm_WARNING+0x28/0x30 [lkdtm 6c2215028606bda50de823490723dc4bc5bf46f9] direct_entry+0x16c/0x1b4 [lkdtm 6c2215028606bda50de823490723dc4bc5bf46f9] full_proxy_write+0x74/0xa4 vfs_write+0xec/0x2e8 ksys_write+0x84/0xf0 __arm64_sys_write+0x24/0x30 el0_svc_common+0xf4/0x1c0 do_el0_svc_compat+0x28/0x3c el0_svc_compat+0x10/0x1c el0_sync_compat_handler+0xa8/0xcc el0_sync_compat+0x178/0x180 ---[ end trace f89bc7f5417cbcc6 ]--- Cc: Jiri Olsa Cc: Alexei Starovoitov Cc: Jessica Yu Cc: Evan Green Cc: Hsin-Yi Wang Cc: Petr Mladek Cc: Steven Rostedt Cc: Sergey Senozhatsky Cc: Andy Shevchenko Cc: Rasmus Villemoes Cc: Cc: Matthew Wilcox Link: https://fedoraproject.org/wiki/Releases/FeatureBuildId [1] Link: https://sourceware.org/elfutils/Debuginfod.html [2] Signed-off-by: Stephen Boyd --- Documentation/core-api/printk-formats.rst | 11 +++ include/linux/kallsyms.h | 20 - include/linux/module.h| 6 +- kernel/kallsyms.c | 95 ++- kernel/module.c | 24 +- lib/vsprintf.c| 8 +- 6 files changed, 139 insertions(+), 25 deletions(-) diff --git a/Documentation/core-api/printk-formats.rst b/Documentation/core-api/printk-formats.rst index 160e710d992f..5f60533f2a56 100644 --- a/Documentation/core-api/printk-formats.rst +++ b/Documentation/core-api/printk-formats.rst @@ -114,6 +114,17 @@ used when printing stack backtraces. The specifier takes into consideration the effect of compiler optimisations which may occur when tail-calls are used and marked with the noreturn GCC attribute. +If the pointer is within a module, the module name and
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
On Tue 2021-04-13 15:57:49, Stephen Boyd wrote:
> Quoting Petr Mladek (2021-04-13 08:01:14)
> > On Fri 2021-04-09 18:52:52, Stephen Boyd wrote:
> > > Let's make kernel stacktraces easier to identify by including the build
> > > ID[1] of a module if the stacktrace is printing a symbol from a module.
> > > This makes it simpler for developers to locate a kernel module's full
> > > debuginfo for a particular stacktrace. Combined with
> > > scripts/decode_stracktrace.sh, a developer can download the matching
> > > debuginfo from a debuginfod[2] server and find the exact file and line
> > > number for the functions plus offsets in a stacktrace that match the
> > > module. This is especially useful for pstore crash debugging where the
> > > kernel crashes are recorded in something like console-ramoops and the
> > > recovery kernel/modules are different or the debuginfo doesn't exist on
> > > the device due to space concerns (the debuginfo can be too large for
> > > space limited devices).
> > >
> > > diff --git a/include/linux/module.h b/include/linux/module.h
> > > index 59f094fa6f74..4bf869f6c944 100644
> > > --- a/include/linux/module.h
> > > +++ b/include/linux/module.h
> > > @@ -11,6 +11,7 @@
> > >
> > > #include
> > > #include
> > > +#include
> > > #include
> > > #include
> > > #include
> > > @@ -367,6 +368,9 @@ struct module {
> > > /* Unique handle for this module */
> > > char name[MODULE_NAME_LEN];
> > >
> > > + /* Module build ID */
> > > + unsigned char build_id[BUILD_ID_SIZE_MAX];
> >
> > Do we want to initialize/store the ID even when
> > CONFIG_STACKTRACE_BUILD_ID is disabled and nobody would
> > use it?
> >
> > Most struct module members are added only when the related feature
> > is enabled.
> >
> > I am not sure how it would complicate the code. It is possible
> > that it is not worth it. Well, I could imagine that the API
> > will always pass the buildid parameter and
> > module_address_lookup() might do something like
> >
> > #ifndef CONFIG_STACKTRACE_BUILD_ID
> > static char empty_build_id[BUILD_ID_SIZE_MAX];
> > #endif
> >
> > if (modbuildid) {
> > if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID))
> > *modbuildid = mod->build_id;
> > else
> > *modbuildid = empty_build_id;
> >
> > IMHO, this is primary a call for Jessica as the module code maintainer.
> >
> > Otherwise, I am fine with this patch. And it works as expected.
> >
>
> Does declaring mod->build_id as zero length work well enough?
It might be fine because it would actually never get displayed.
But yeah, it is kind of hack. The idea was to avoid too many
#ifdefs in the code.
I think that it is Jessica's call what she would prefer.
Best Regards,
Petr
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
Quoting Petr Mladek (2021-04-13 08:01:14)
> On Fri 2021-04-09 18:52:52, Stephen Boyd wrote:
> > Let's make kernel stacktraces easier to identify by including the build
> > ID[1] of a module if the stacktrace is printing a symbol from a module.
> > This makes it simpler for developers to locate a kernel module's full
> > debuginfo for a particular stacktrace. Combined with
> > scripts/decode_stracktrace.sh, a developer can download the matching
> > debuginfo from a debuginfod[2] server and find the exact file and line
> > number for the functions plus offsets in a stacktrace that match the
> > module. This is especially useful for pstore crash debugging where the
> > kernel crashes are recorded in something like console-ramoops and the
> > recovery kernel/modules are different or the debuginfo doesn't exist on
> > the device due to space concerns (the debuginfo can be too large for
> > space limited devices).
> >
> > diff --git a/include/linux/module.h b/include/linux/module.h
> > index 59f094fa6f74..4bf869f6c944 100644
> > --- a/include/linux/module.h
> > +++ b/include/linux/module.h
> > @@ -11,6 +11,7 @@
> >
> > #include
> > #include
> > +#include
> > #include
> > #include
> > #include
> > @@ -367,6 +368,9 @@ struct module {
> > /* Unique handle for this module */
> > char name[MODULE_NAME_LEN];
> >
> > + /* Module build ID */
> > + unsigned char build_id[BUILD_ID_SIZE_MAX];
>
> Do we want to initialize/store the ID even when
> CONFIG_STACKTRACE_BUILD_ID is disabled and nobody would
> use it?
>
> Most struct module members are added only when the related feature
> is enabled.
>
> I am not sure how it would complicate the code. It is possible
> that it is not worth it. Well, I could imagine that the API
> will always pass the buildid parameter and
> module_address_lookup() might do something like
>
> #ifndef CONFIG_STACKTRACE_BUILD_ID
> static char empty_build_id[BUILD_ID_SIZE_MAX];
> #endif
>
> if (modbuildid) {
> if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID))
> *modbuildid = mod->build_id;
> else
> *modbuildid = empty_build_id;
>
> IMHO, this is primary a call for Jessica as the module code maintainer.
>
> Otherwise, I am fine with this patch. And it works as expected.
>
Does declaring mod->build_id as zero length work well enough?
8<
diff --git a/include/linux/module.h b/include/linux/module.h
index 4bf869f6c944..03b2f6af093a 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -359,6 +359,12 @@ struct klp_modinfo {
};
#endif
+#if IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID)
+#define MODULE_BUILD_ID_LEN BUILD_ID_SIZE_MAX
+#else
+#define MODULE_BUILD_ID_LEN 0
+#endif
+
struct module {
enum module_state state;
@@ -369,7 +375,7 @@ struct module {
char name[MODULE_NAME_LEN];
/* Module build ID */
- unsigned char build_id[BUILD_ID_SIZE_MAX];
+ unsigned char build_id[MODULE_BUILD_ID_LEN];
/* Sysfs stuff. */
struct module_kobject mkobj;
diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index b835992e76c2..ebd5b30c3039 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -25,7 +25,10 @@
#include
#include
#include
+#include
#include
+#include
+#include
/*
* These will be re-linked against their real values
@@ -393,10 +396,13 @@ static int __sprint_symbol(char *buffer, unsigned long
address,
if (modname) {
len += sprintf(buffer + len, " [%s", modname);
- /* build ID should match length of sprintf below */
- BUILD_BUG_ON(BUILD_ID_SIZE_MAX != 20);
- if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) && add_buildid &&
buildid)
+#if IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID)
+ if (add_buildid && buildid) {
+ /* build ID should match length of sprintf */
+ static_assert(MODULE_BUILD_ID_LEN == 20);
len += sprintf(buffer + len, " %20phN", buildid);
+ }
+#endif
len += sprintf(buffer + len, "]");
}
diff --git a/kernel/module.c b/kernel/module.c
index 6f5bc1b046a5..a0d222fbd281 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -2771,7 +2771,17 @@ static void add_kallsyms(struct module *mod, const
struct load_info *info)
}
mod->core_kallsyms.num_symtab = ndst;
}
+#else
+static inline void layout_symtab(struct module *mod, struct load_info *info)
+{
+}
+
+static void add_kallsyms(struct module *mod, const struct load_info *info)
+{
+}
+#endif /* CONFIG_KALLSYMS */
+#if IS_ENABLED(CONFIG_KALLSYMS) && IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID)
static void init_build_id(struct module *mod, const struct load_info *info)
{
const Elf_Shdr *sechdr;
@@ -2786,18 +2796,10 @@ static void init_build_id(struct module *mod, const
struct load_info *info)
}
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
Quoting Stephen Boyd (2021-04-13 13:10:05)
> Quoting Petr Mladek (2021-04-13 08:16:20)
> > On Tue 2021-04-13 13:56:31, Andy Shevchenko wrote:
> > > On Mon, Apr 12, 2021 at 12:29:05PM -0700, Stephen Boyd wrote:
> > > > Quoting Andy Shevchenko (2021-04-12 04:58:02)
> > > > >
> > > > > First of all, why not static_assert() defined near to the actual
> > > > > macro?
> > > >
> > > > Which macro? BUILD_ID_SIZE_MAX?
> > >
> > > Yes.
> > >
> > > > I tried static_assert() and it didn't
> > > > work for me but maybe I missed something.
> >
> > I guess that you wanted to use it inside macro definition:
> >
> > #define VMCOREINFO_BUILD_ID(value) \
> > static_assert(ARRAY_SIZE(value) == BUILD_ID_SIZE_MAX); \
> > vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
> >
> > Instead, you should do it outside the macro:
> >
> > static_assert(ARRAY_SIZE(value) == BUILD_ID_SIZE_MAX);
> > #define VMCOREINFO_BUILD_ID(value) \
> > vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
>
> In this example "value" is not defined because it's an argument to the
> macro. How can this work?
>
> From what I can tell static_assert() is for the case that you want to
> assert something at the global scope level. BUILD_BUG_ON() can't be used
> at global scope. I see the usage is usually to assert struct members and
> alignment of those members. In turn, static_assert() can't be used at
> function level scope. Each has a use and in this case I want to assert
> at function level scope to be as close as possible to the place that
> would need to change.
>
Good news. I can do this to force a basic block and then GCC doesn't complain.
---8<---
diff --git a/include/linux/crash_core.h b/include/linux/crash_core.h
index 2174dab16ba9..de62a722431e 100644
--- a/include/linux/crash_core.h
+++ b/include/linux/crash_core.h
@@ -38,9 +38,12 @@ phys_addr_t paddr_vmcoreinfo_note(void);
#define VMCOREINFO_OSRELEASE(value) \
vmcoreinfo_append_str("OSRELEASE=%s\n", value)
-#define VMCOREINFO_BUILD_ID(value) \
- BUILD_BUG_ON(ARRAY_SIZE(value) != BUILD_ID_SIZE_MAX); \
- vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
+#define VMCOREINFO_BUILD_ID() \
+ ({ \
+ static_assert(sizeof(vmlinux_build_id) == 20); \
+ vmcoreinfo_append_str("BUILD-ID=%20phN\n", vmlinux_build_id); \
+ })
+
#define VMCOREINFO_PAGESIZE(value) \
vmcoreinfo_append_str("PAGESIZE=%ld\n", value)
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
Quoting Petr Mladek (2021-04-13 08:16:20)
> On Tue 2021-04-13 13:56:31, Andy Shevchenko wrote:
> > On Mon, Apr 12, 2021 at 12:29:05PM -0700, Stephen Boyd wrote:
> > > Quoting Andy Shevchenko (2021-04-12 04:58:02)
> > > >
> > > > First of all, why not static_assert() defined near to the actual macro?
> > >
> > > Which macro? BUILD_ID_SIZE_MAX?
> >
> > Yes.
> >
> > > I tried static_assert() and it didn't
> > > work for me but maybe I missed something.
>
> I guess that you wanted to use it inside macro definition:
>
> #define VMCOREINFO_BUILD_ID(value) \
> static_assert(ARRAY_SIZE(value) == BUILD_ID_SIZE_MAX); \
> vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
>
> Instead, you should do it outside the macro:
>
> static_assert(ARRAY_SIZE(value) == BUILD_ID_SIZE_MAX);
> #define VMCOREINFO_BUILD_ID(value) \
> vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
In this example "value" is not defined because it's an argument to the
macro. How can this work?
>From what I can tell static_assert() is for the case that you want to
assert something at the global scope level. BUILD_BUG_ON() can't be used
at global scope. I see the usage is usually to assert struct members and
alignment of those members. In turn, static_assert() can't be used at
function level scope. Each has a use and in this case I want to assert
at function level scope to be as close as possible to the place that
would need to change.
>
> > Sounds weird. static_assert() is a good one. Check, for example,
> > lib/vsprintf.c
> > on how to use it.
> >
> > > Why is static_assert()
> > > preferred?
>
> I guess that it is because it is enough and more efficient for
> checks of constant values (no computation of the value).
>
> > Because it's cleaner way to achieve it and as a bonus it can be put outside
> > of
> > the functions (be in the header or so).
Ok, but I'm still not sure what it would be enforcing. In this case we
need to have it near the sprintf line so that we know to fix the 20 in
there should it ever change to be larger. If it's defined next to the
BUILD_ID_SIZE_MAX macro then it does practically nothing to help future
developers know what to change.
> >
> > > > > + if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) &&
> > > > > add_buildid && buildid)
> > > > > + len += sprintf(buffer + len, " %20phN",
> > > > > buildid);
> > > >
> > > > len += sprintf(buffer + len, " %*phN",
> > > > BUILD_ID_SIZE_MAX, buildid);
> > > >
> > >
> > > Are you suggesting to use sprintf format here so that the size is part
> > > of the printf? Sounds good to me. Thanks.
> >
> > I prefer %20phN when the size is carved in stone (for example by
> > specification), but if you are really expecting that it may be
> > changed in the future, use variadic approach as I showed above.
>
> I would consider this written in stone (last famous words ;-) and use
> %20phN with the static_assert().
>
Yes it is pretty much written in stone. The build ID can be an md5sum
instead of SHA1, and thus 16 bytes instead of 20 bytes for the 160-bit
SHA1 form. This is rare, and the code in buildid.c is padding it out
with zeroes in the case that the note is smaller than 20 bytes in
length. Within the kernel we can always assume the buffer is
BUILD_ID_SIZE_MAX. How about this patch?
8<-
diff --git a/include/linux/crash_core.h b/include/linux/crash_core.h
index 2174dab16ba9..042c9c034fba 100644
--- a/include/linux/crash_core.h
+++ b/include/linux/crash_core.h
@@ -39,7 +39,7 @@ phys_addr_t paddr_vmcoreinfo_note(void);
#define VMCOREINFO_OSRELEASE(value) \
vmcoreinfo_append_str("OSRELEASE=%s\n", value)
#define VMCOREINFO_BUILD_ID(value) \
- BUILD_BUG_ON(ARRAY_SIZE(value) != BUILD_ID_SIZE_MAX); \
+ BUILD_BUG_ON(ARRAY_SIZE(value) != 20); \
vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
#define VMCOREINFO_PAGESIZE(value) \
vmcoreinfo_append_str("PAGESIZE=%ld\n", value)
diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index b835992e76c2..5d9c7ac80633 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -25,7 +25,10 @@
#include
#include
#include
+#include
#include
+#include
+#include
/*
* These will be re-linked against their real values
@@ -394,7 +397,7 @@ static int __sprint_symbol(char *buffer, unsigned long
address,
if (modname) {
len += sprintf(buffer + len, " [%s", modname);
/* build ID should match length of sprintf below */
- BUILD_BUG_ON(BUILD_ID_SIZE_MAX != 20);
+ BUILD_BUG_ON(sizeof(typeof_member(struct module, build_id)) !=
20);
if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) && add_buildid &&
buildid)
len += sprintf(buffer + len, " %20phN", buildid);
len += sprintf(buffer + len, "]");
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
On Tue 2021-04-13 13:56:31, Andy Shevchenko wrote:
> On Mon, Apr 12, 2021 at 12:29:05PM -0700, Stephen Boyd wrote:
> > Quoting Andy Shevchenko (2021-04-12 04:58:02)
> > > On Fri, Apr 09, 2021 at 06:52:52PM -0700, Stephen Boyd wrote:
> > > > Let's make kernel stacktraces easier to identify by including the build
> > > > ID[1] of a module if the stacktrace is printing a symbol from a module.
> > > > This makes it simpler for developers to locate a kernel module's full
> > > > debuginfo for a particular stacktrace. Combined with
> > > > scripts/decode_stracktrace.sh, a developer can download the matching
> > > > debuginfo from a debuginfod[2] server and find the exact file and line
> > > > number for the functions plus offsets in a stacktrace that match the
> > > > module. This is especially useful for pstore crash debugging where the
> > > > kernel crashes are recorded in something like console-ramoops and the
> > > > recovery kernel/modules are different or the debuginfo doesn't exist on
> > > > the device due to space concerns (the debuginfo can be too large for
> > > > space limited devices).
> > > >
> > > > Originally, I put this on the %pS format, but that was quickly rejected
> > > > given that %pS is used in other places such as ftrace where build IDs
> > > > aren't meaningful. There was some discussions on the list to put every
> > > > module build ID into the "Modules linked in:" section of the stacktrace
> > > > message but that quickly becomes very hard to read once you have more
> > > > than three or four modules linked in. It also provides too much
> > > > information when we don't expect each module to be traversed in a
> > > > stacktrace. Having the build ID for modules that aren't important just
> > > > makes things messy. Splitting it to multiple lines for each module
> > > > quickly explodes the number of lines printed in an oops too, possibly
> > > > wrapping the warning off the console. And finally, trying to stash away
> > > > each module used in a callstack to provide the ID of each symbol printed
> > > > is cumbersome and would require changes to each architecture to stash
> > > > away modules and return their build IDs once unwinding has completed.
> > > >
> > > > Instead, we opt for the simpler approach of introducing new printk
> > > > formats '%pS[R]b' for "pointer symbolic backtrace with module build ID"
> > > > and '%pBb' for "pointer backtrace with module build ID" and then
> > > > updating the few places in the architecture layer where the stacktrace
> > > > is printed to use this new format.
> > > >
> > > > Example:
> > >
> > > Can you trim a bit the example, so we will see only important lines.
> > > In such case you may provide "before" and "after" variants.
> > >
> > > ...
> > >
> > > > - if (modname)
> > > > - len += sprintf(buffer + len, " [%s]", modname);
> > > > + if (modname) {
> > > > + len += sprintf(buffer + len, " [%s", modname);
> > >
> > > > + /* build ID should match length of sprintf below */
> > > > + BUILD_BUG_ON(BUILD_ID_SIZE_MAX != 20);
> > >
> > > First of all, why not static_assert() defined near to the actual macro?
> >
> > Which macro? BUILD_ID_SIZE_MAX?
>
> Yes.
>
> > I tried static_assert() and it didn't
> > work for me but maybe I missed something.
I guess that you wanted to use it inside macro definition:
#define VMCOREINFO_BUILD_ID(value) \
static_assert(ARRAY_SIZE(value) == BUILD_ID_SIZE_MAX); \
vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
Instead, you should do it outside the macro:
static_assert(ARRAY_SIZE(value) == BUILD_ID_SIZE_MAX);
#define VMCOREINFO_BUILD_ID(value) \
vmcoreinfo_append_str("BUILD-ID=%20phN\n", value)
> Sounds weird. static_assert() is a good one. Check, for example,
> lib/vsprintf.c
> on how to use it.
>
> > Why is static_assert()
> > preferred?
I guess that it is because it is enough and more efficient for
checks of constant values (no computation of the value).
> Because it's cleaner way to achieve it and as a bonus it can be put outside of
> the functions (be in the header or so).
>
> > > > + if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) && add_buildid
> > > > && buildid)
> > > > + len += sprintf(buffer + len, " %20phN", buildid);
> > >
> > > len += sprintf(buffer + len, " %*phN",
> > > BUILD_ID_SIZE_MAX, buildid);
> > >
> >
> > Are you suggesting to use sprintf format here so that the size is part
> > of the printf? Sounds good to me. Thanks.
>
> I prefer %20phN when the size is carved in stone (for example by
> specification), but if you are really expecting that it may be
> changed in the future, use variadic approach as I showed above.
I would consider this written in stone (last famous words ;-) and use
%20phN with the static_assert().
Best Regards,
Petr
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
On Fri 2021-04-09 18:52:52, Stephen Boyd wrote:
> Let's make kernel stacktraces easier to identify by including the build
> ID[1] of a module if the stacktrace is printing a symbol from a module.
> This makes it simpler for developers to locate a kernel module's full
> debuginfo for a particular stacktrace. Combined with
> scripts/decode_stracktrace.sh, a developer can download the matching
> debuginfo from a debuginfod[2] server and find the exact file and line
> number for the functions plus offsets in a stacktrace that match the
> module. This is especially useful for pstore crash debugging where the
> kernel crashes are recorded in something like console-ramoops and the
> recovery kernel/modules are different or the debuginfo doesn't exist on
> the device due to space concerns (the debuginfo can be too large for
> space limited devices).
>
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 59f094fa6f74..4bf869f6c944 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -11,6 +11,7 @@
>
> #include
> #include
> +#include
> #include
> #include
> #include
> @@ -367,6 +368,9 @@ struct module {
> /* Unique handle for this module */
> char name[MODULE_NAME_LEN];
>
> + /* Module build ID */
> + unsigned char build_id[BUILD_ID_SIZE_MAX];
Do we want to initialize/store the ID even when
CONFIG_STACKTRACE_BUILD_ID is disabled and nobody would
use it?
Most struct module members are added only when the related feature
is enabled.
I am not sure how it would complicate the code. It is possible
that it is not worth it. Well, I could imagine that the API
will always pass the buildid parameter and
module_address_lookup() might do something like
#ifndef CONFIG_STACKTRACE_BUILD_ID
static char empty_build_id[BUILD_ID_SIZE_MAX];
#endif
if (modbuildid) {
if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID))
*modbuildid = mod->build_id;
else
*modbuildid = empty_build_id;
IMHO, this is primary a call for Jessica as the module code maintainer.
Otherwise, I am fine with this patch. And it works as expected.
Best Regards,
Petr
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
On Mon, Apr 12, 2021 at 12:29:05PM -0700, Stephen Boyd wrote:
> Quoting Andy Shevchenko (2021-04-12 04:58:02)
> > On Fri, Apr 09, 2021 at 06:52:52PM -0700, Stephen Boyd wrote:
> > > Let's make kernel stacktraces easier to identify by including the build
> > > ID[1] of a module if the stacktrace is printing a symbol from a module.
> > > This makes it simpler for developers to locate a kernel module's full
> > > debuginfo for a particular stacktrace. Combined with
> > > scripts/decode_stracktrace.sh, a developer can download the matching
> > > debuginfo from a debuginfod[2] server and find the exact file and line
> > > number for the functions plus offsets in a stacktrace that match the
> > > module. This is especially useful for pstore crash debugging where the
> > > kernel crashes are recorded in something like console-ramoops and the
> > > recovery kernel/modules are different or the debuginfo doesn't exist on
> > > the device due to space concerns (the debuginfo can be too large for
> > > space limited devices).
> > >
> > > Originally, I put this on the %pS format, but that was quickly rejected
> > > given that %pS is used in other places such as ftrace where build IDs
> > > aren't meaningful. There was some discussions on the list to put every
> > > module build ID into the "Modules linked in:" section of the stacktrace
> > > message but that quickly becomes very hard to read once you have more
> > > than three or four modules linked in. It also provides too much
> > > information when we don't expect each module to be traversed in a
> > > stacktrace. Having the build ID for modules that aren't important just
> > > makes things messy. Splitting it to multiple lines for each module
> > > quickly explodes the number of lines printed in an oops too, possibly
> > > wrapping the warning off the console. And finally, trying to stash away
> > > each module used in a callstack to provide the ID of each symbol printed
> > > is cumbersome and would require changes to each architecture to stash
> > > away modules and return their build IDs once unwinding has completed.
> > >
> > > Instead, we opt for the simpler approach of introducing new printk
> > > formats '%pS[R]b' for "pointer symbolic backtrace with module build ID"
> > > and '%pBb' for "pointer backtrace with module build ID" and then
> > > updating the few places in the architecture layer where the stacktrace
> > > is printed to use this new format.
> > >
> > > Example:
> >
> > Can you trim a bit the example, so we will see only important lines.
> > In such case you may provide "before" and "after" variants.
> >
> > ...
> >
> > > - if (modname)
> > > - len += sprintf(buffer + len, " [%s]", modname);
> > > + if (modname) {
> > > + len += sprintf(buffer + len, " [%s", modname);
> >
> > > + /* build ID should match length of sprintf below */
> > > + BUILD_BUG_ON(BUILD_ID_SIZE_MAX != 20);
> >
> > First of all, why not static_assert() defined near to the actual macro?
>
> Which macro? BUILD_ID_SIZE_MAX?
Yes.
> I tried static_assert() and it didn't
> work for me but maybe I missed something.
Sounds weird. static_assert() is a good one. Check, for example, lib/vsprintf.c
on how to use it.
> Why is static_assert()
> preferred?
Because it's cleaner way to achieve it and as a bonus it can be put outside of
the functions (be in the header or so).
> > > + if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) && add_buildid
> > > && buildid)
> > > + len += sprintf(buffer + len, " %20phN", buildid);
> >
> > len += sprintf(buffer + len, " %*phN",
> > BUILD_ID_SIZE_MAX, buildid);
> >
>
> Are you suggesting to use sprintf format here so that the size is part
> of the printf? Sounds good to me. Thanks.
I prefer %20phN when the size is carved in stone (for example by
specification), but if you are really expecting that it may be
changed in the future, use variadic approach as I showed above.
--
With Best Regards,
Andy Shevchenko
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
Quoting Andy Shevchenko (2021-04-12 04:58:02)
> On Fri, Apr 09, 2021 at 06:52:52PM -0700, Stephen Boyd wrote:
> > Let's make kernel stacktraces easier to identify by including the build
> > ID[1] of a module if the stacktrace is printing a symbol from a module.
> > This makes it simpler for developers to locate a kernel module's full
> > debuginfo for a particular stacktrace. Combined with
> > scripts/decode_stracktrace.sh, a developer can download the matching
> > debuginfo from a debuginfod[2] server and find the exact file and line
> > number for the functions plus offsets in a stacktrace that match the
> > module. This is especially useful for pstore crash debugging where the
> > kernel crashes are recorded in something like console-ramoops and the
> > recovery kernel/modules are different or the debuginfo doesn't exist on
> > the device due to space concerns (the debuginfo can be too large for
> > space limited devices).
> >
> > Originally, I put this on the %pS format, but that was quickly rejected
> > given that %pS is used in other places such as ftrace where build IDs
> > aren't meaningful. There was some discussions on the list to put every
> > module build ID into the "Modules linked in:" section of the stacktrace
> > message but that quickly becomes very hard to read once you have more
> > than three or four modules linked in. It also provides too much
> > information when we don't expect each module to be traversed in a
> > stacktrace. Having the build ID for modules that aren't important just
> > makes things messy. Splitting it to multiple lines for each module
> > quickly explodes the number of lines printed in an oops too, possibly
> > wrapping the warning off the console. And finally, trying to stash away
> > each module used in a callstack to provide the ID of each symbol printed
> > is cumbersome and would require changes to each architecture to stash
> > away modules and return their build IDs once unwinding has completed.
> >
> > Instead, we opt for the simpler approach of introducing new printk
> > formats '%pS[R]b' for "pointer symbolic backtrace with module build ID"
> > and '%pBb' for "pointer backtrace with module build ID" and then
> > updating the few places in the architecture layer where the stacktrace
> > is printed to use this new format.
> >
> > Example:
>
> Can you trim a bit the example, so we will see only important lines.
> In such case you may provide "before" and "after" variants.
>
> ...
>
> > - if (modname)
> > - len += sprintf(buffer + len, " [%s]", modname);
> > + if (modname) {
> > + len += sprintf(buffer + len, " [%s", modname);
>
> > + /* build ID should match length of sprintf below */
> > + BUILD_BUG_ON(BUILD_ID_SIZE_MAX != 20);
>
> First of all, why not static_assert() defined near to the actual macro?
Which macro? BUILD_ID_SIZE_MAX? I tried static_assert() and it didn't
work for me but maybe I missed something. Why is static_assert()
preferred?
>
> > + if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) && add_buildid &&
> > buildid)
> > + len += sprintf(buffer + len, " %20phN", buildid);
>
> len += sprintf(buffer + len, " %*phN",
> BUILD_ID_SIZE_MAX, buildid);
>
Are you suggesting to use sprintf format here so that the size is part
of the printf? Sounds good to me. Thanks.
Re: [PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
On Fri, Apr 09, 2021 at 06:52:52PM -0700, Stephen Boyd wrote:
> Let's make kernel stacktraces easier to identify by including the build
> ID[1] of a module if the stacktrace is printing a symbol from a module.
> This makes it simpler for developers to locate a kernel module's full
> debuginfo for a particular stacktrace. Combined with
> scripts/decode_stracktrace.sh, a developer can download the matching
> debuginfo from a debuginfod[2] server and find the exact file and line
> number for the functions plus offsets in a stacktrace that match the
> module. This is especially useful for pstore crash debugging where the
> kernel crashes are recorded in something like console-ramoops and the
> recovery kernel/modules are different or the debuginfo doesn't exist on
> the device due to space concerns (the debuginfo can be too large for
> space limited devices).
>
> Originally, I put this on the %pS format, but that was quickly rejected
> given that %pS is used in other places such as ftrace where build IDs
> aren't meaningful. There was some discussions on the list to put every
> module build ID into the "Modules linked in:" section of the stacktrace
> message but that quickly becomes very hard to read once you have more
> than three or four modules linked in. It also provides too much
> information when we don't expect each module to be traversed in a
> stacktrace. Having the build ID for modules that aren't important just
> makes things messy. Splitting it to multiple lines for each module
> quickly explodes the number of lines printed in an oops too, possibly
> wrapping the warning off the console. And finally, trying to stash away
> each module used in a callstack to provide the ID of each symbol printed
> is cumbersome and would require changes to each architecture to stash
> away modules and return their build IDs once unwinding has completed.
>
> Instead, we opt for the simpler approach of introducing new printk
> formats '%pS[R]b' for "pointer symbolic backtrace with module build ID"
> and '%pBb' for "pointer backtrace with module build ID" and then
> updating the few places in the architecture layer where the stacktrace
> is printed to use this new format.
>
> Example:
Can you trim a bit the example, so we will see only important lines.
In such case you may provide "before" and "after" variants.
...
> - if (modname)
> - len += sprintf(buffer + len, " [%s]", modname);
> + if (modname) {
> + len += sprintf(buffer + len, " [%s", modname);
> + /* build ID should match length of sprintf below */
> + BUILD_BUG_ON(BUILD_ID_SIZE_MAX != 20);
First of all, why not static_assert() defined near to the actual macro?
> + if (IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) && add_buildid &&
> buildid)
> + len += sprintf(buffer + len, " %20phN", buildid);
len += sprintf(buffer + len, " %*phN",
BUILD_ID_SIZE_MAX, buildid);
?
--
With Best Regards,
Andy Shevchenko
[PATCH v4 05/13] module: Add printk formats to add module build ID to stacktraces
Let's make kernel stacktraces easier to identify by including the build ID[1] of a module if the stacktrace is printing a symbol from a module. This makes it simpler for developers to locate a kernel module's full debuginfo for a particular stacktrace. Combined with scripts/decode_stracktrace.sh, a developer can download the matching debuginfo from a debuginfod[2] server and find the exact file and line number for the functions plus offsets in a stacktrace that match the module. This is especially useful for pstore crash debugging where the kernel crashes are recorded in something like console-ramoops and the recovery kernel/modules are different or the debuginfo doesn't exist on the device due to space concerns (the debuginfo can be too large for space limited devices). Originally, I put this on the %pS format, but that was quickly rejected given that %pS is used in other places such as ftrace where build IDs aren't meaningful. There was some discussions on the list to put every module build ID into the "Modules linked in:" section of the stacktrace message but that quickly becomes very hard to read once you have more than three or four modules linked in. It also provides too much information when we don't expect each module to be traversed in a stacktrace. Having the build ID for modules that aren't important just makes things messy. Splitting it to multiple lines for each module quickly explodes the number of lines printed in an oops too, possibly wrapping the warning off the console. And finally, trying to stash away each module used in a callstack to provide the ID of each symbol printed is cumbersome and would require changes to each architecture to stash away modules and return their build IDs once unwinding has completed. Instead, we opt for the simpler approach of introducing new printk formats '%pS[R]b' for "pointer symbolic backtrace with module build ID" and '%pBb' for "pointer backtrace with module build ID" and then updating the few places in the architecture layer where the stacktrace is printed to use this new format. Example: WARNING: CPU: 3 PID: 3373 at drivers/misc/lkdtm/bugs.c:83 lkdtm_WARNING+0x28/0x30 [lkdtm] Modules linked in: lkdtm rfcomm algif_hash algif_skcipher af_alg xt_cgroup uinput xt_MASQUERADE hci_uart CPU: 3 PID: 3373 Comm: bash Not tainted 5.11 #12 a8c0d47f7051f3e6670ceaea724af66a39c6cec8 Hardware name: Google Lazor (rev3+) with KB Backlight (DT) pstate: 0049 (nzcv daif +PAN -UAO -TCO BTYPE=--) pc : lkdtm_WARNING+0x28/0x30 [lkdtm] lr : lkdtm_do_action+0x24/0x40 [lkdtm] sp : ffc013febca0 x29: ffc013febca0 x28: ff88d9438040 x27: x26: x25: x24: ffdd0e9772c0 x23: 0020 x22: ffdd0e975366 x21: ffdd0e9772e0 x20: ffc013febde0 x19: 0008 x18: x17: x16: 0037 x15: ffdd102ab174 x14: 0003 x13: 0004 x12: x11: x10: x9 : 0001 x8 : ffdd0e979000 x7 : x6 : ffdd10ff6b54 x5 : x4 : x3 : ffc013feb938 x2 : ff89fef05a70 x1 : ff89feef5788 x0 : ffdd0e9772e0 Call trace: lkdtm_WARNING+0x28/0x30 [lkdtm 6c2215028606bda50de823490723dc4bc5bf46f9] direct_entry+0x16c/0x1b4 [lkdtm 6c2215028606bda50de823490723dc4bc5bf46f9] full_proxy_write+0x74/0xa4 vfs_write+0xec/0x2e8 ksys_write+0x84/0xf0 __arm64_sys_write+0x24/0x30 el0_svc_common+0xf4/0x1c0 do_el0_svc_compat+0x28/0x3c el0_svc_compat+0x10/0x1c el0_sync_compat_handler+0xa8/0xcc el0_sync_compat+0x178/0x180 ---[ end trace f89bc7f5417cbcc6 ]--- Cc: Jiri Olsa Cc: Alexei Starovoitov Cc: Jessica Yu Cc: Evan Green Cc: Hsin-Yi Wang Cc: Petr Mladek Cc: Steven Rostedt Cc: Sergey Senozhatsky Cc: Andy Shevchenko Cc: Rasmus Villemoes Cc: Cc: Matthew Wilcox Link: https://fedoraproject.org/wiki/Releases/FeatureBuildId [1] Link: https://sourceware.org/elfutils/Debuginfod.html [2] Signed-off-by: Stephen Boyd --- Documentation/core-api/printk-formats.rst | 11 +++ include/linux/kallsyms.h | 20 - include/linux/module.h| 6 +- kernel/kallsyms.c | 95 ++- kernel/module.c | 24 +- lib/vsprintf.c| 8 +- 6 files changed, 139 insertions(+), 25 deletions(-) diff --git a/Documentation/core-api/printk-formats.rst b/Documentation/core-api/printk-formats.rst index 160e710d992f..5f60533f2a56 100644 --- a/Documentation/core-api/printk-formats.rst +++ b/Documentation/core-api/printk-formats.rst @@ -114,6 +114,17 @@ used when printing stack backtraces. The specifier takes into consideration the effect of compiler optimisations which may occur when tail-calls are used and marked with the noreturn GCC attribute. +If the pointer is within a module, the module name
