Hi Tao, I like the idea of having just one thread at a time from the gdb side, and it can be replaced by a "set" crash command. Single semantics will work for both live and offline debugging and for both types of tasks: active or scheduled out. Need to be careful with lifetime for the data (gdb caches). I'm experimenting with live debugging these days. Aditya, Tao, let me know what is the latest patchset I should use from you folks?
Thanks, --Alexey On Mon, Mar 18, 2024 at 2:56 AM Tao Liu <l...@redhat.com> wrote: > Hi Aditya and Alexey, > > On Mon, Mar 18, 2024 at 11:35:41AM +0800, Tao Liu wrote: > > On Sun, Mar 17, 2024 at 03:40:52PM +0530, Aditya Gupta wrote: > > > Hi Tao and Alexey, > > > > > > On Fri, Mar 15, 2024 at 03:15:19PM -0700, Alexey Makhalov wrote: > > > > Hi Tao, > > > > > > > > Thanks for the details. Agree, my approach is targeted primarily for > > > > offline debugging. > > > > Create/exit monitoring sounds like an overkill. And from the other > side, > > > > maintaining a static list of threads (CPUs or Tasks) does not work > for live > > > > debugging. > > > > "ps" followed by "set" where GDB will receive just one > inferior/thread at a > > > > time is probably the right universal approach. > > > > Let me experiment with a combination of my and your patches and come > back > > > > to you. > > > > > > Sorry I noticed a bug late while trying the combination of tao's and my > > > patches. > > > > > > I agree with Tao that current approach might not be good for live > > > debugging, but I think the approach of having all tasks in 'info > > > threads' is a good advantage for offline debugging, since then gdb > knows > > > about all tasks (similar behaviour as usual gdb), and makes it easier > > > to synchronise crash <-> gdb context. > > > > > > I had refrained from changing the format of 'info threads' from active > > > tasks to all tasks, but there are some issues with crash <-> gdb sync, > > > when doing arbitrary task unwinding. > > > > > > 'set 1' (let's assume it's an INACTIVE task on CPU 5) > > > > > > 'info threads' > > > -> when it switches to CPU 5, it asks crash to sync, ie. `set_cpu(5)` > > > -> crash will return CPU 5's active tasks registers > > > -> WRONG registers for CPU 5, since we wanted registers for PID 1 > (which > > > is an inactive task on CPU 5) > > > > What do you mean of "WRONG registers"? Isn't the CPU5's regcache be pid > > 1's context after 'set 1'? If it isn't then it's a bug. > > > > After crash finished loading and we see crash> prompt, the cpu regcaches > > will be all active tasks', which are running on each cpu. That is what > we saw > > using "info threads", like: > > > > crash> info threads > > Id Target Id Frame > > .... > > * 8 CPU 7 blk_mq_rq_timed_out > (req=0xffff880fdb246000, reserved=reserved@entry=false) at > block/blk-mq.c:640 > > .... > > 14 CPU 13 native_safe_halt () at > arch/x86/include/asm/irqflags.h:54 > > .... > > > > Then we use "set <pid>" to switch to task <pid>, its cpu regcache will be > > flushed, and if we use command "info threads" again, it will show the > status > > after the flushing. E.g in the following, task pid 1's cpu is #13, so > > cpu #13 is flushed after "set 1": > > > > crash> ps > > PID PPID CPU TASK ST %MEM VSZ RSS > COMM > > 1 0 13 ffff880169b30000 IN 0.0 193052 4180 > systemd > > > > crash> set 1 > > crash> info threads > > Id Target Id Frame > > .... > > 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, > reserved=reserved@entry=false) at block/blk-mq.c:640 > > .... > > * 14 CPU 13 0xffffffff816a8f65 in context_switch > (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... > > .... > > > > When design, I assume the CPUs/threads are like power slots, its number > is > > constant, but the contents can be changed. We don't need to prepare the > > "slots" as many as our tasks quantity (I don't know the maxinum threads > that > > a gdb can accept, e.g. on a system which have 65K tasks, but for most > > tasks we'd never switch to it in order to view its stack trace, I guess > > it would be a waste to call add_thread_silent (target, ...) 65K times). > > > > Currently the gdb CPUs/threads(slots) equals to the system's cpu number. > But > > if a system which have thousands of cpus, which we think is unnecessary > to > > call add_thread_silent 1K times, we can even shrink the slots to be > less. > > > > I made a trial patch of leaving only 1 gdb CPUs/threads(slots) in [1]. > And it can also view arbitrary tasks stack unwinding. In this case, > "info threads" and "thread XX" will have no meaning to be kept. > > Please see my test results: > > KERNEL: /home/ltao/kernel/linux-next-next-20220914/vmlinux > DUMPFILE: /proc/kcore > CPUS: 4 > ^^^^^^^ there are 4 cpus in the system > crash> ps > PID PPID CPU TASK ST %MEM VSZ RSS COMM > ... > 1 0 1 ffff9908002a6500 IN 0.1 253024 19444 > systemd > ... > 5 2 0 ffff9908002a0000 ID 0.0 0 0 > [netns] > ... > ^^^^^^^^^^^^^ we use pid 1 and pid 5, which belongs to 2 different > cpus > crash> set 1 > crash> gdb bt > #0 context_switch (rf=0xffffbf750001bd68, next=0xffff99080032cbc0, > prev=0x0, rq=0xffff990b6fcafe80) at kernel/sched/core.c:5201 > #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510 > #2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586 > #3 0xffffffffb3cf6ab9 in schedule_hrtimeout_range_clock > (expires=expires@entry=0x0, delta=delta@entry=0, > mode=mode@entry=HRTIMER_MODE_ABS, > clock_id=clock_id@entry=1) at kernel/time/hrtimer.c:2296 > ... > crash> bt > PID: 1 TASK: ffff9908002a6500 CPU: 1 COMMAND: "systemd" > #0 [ffffbf750001bdb0] schedule at ffffffffb3cf0603 > #1 [ffffbf750001bdc0] schedule_hrtimeout_range_clock at ffffffffb3cf6ab9 > ... > ^^^ we can view pid 1's stack trace with no problem > crash> info threads > Id Target Id Frame > * 1 CPU 0 context_switch (rf=0xffffbf750001bd68, > next=0xffffffffb4c18a00 <init_task>, prev=0x0, rq=0xffff990b6fc2fe80) at > kernel/sched/core.c:5201 > > crash> set 5 > crash> gdb bt > #0 context_switch (rf=0xffffbf750003be58, next=0xffff9908002d6500, > prev=0x0, rq=0xffff990b6fc2fe80) at kernel/sched/core.c:5201 > #1 __schedule (sched_mode=sched_mode@entry=0) at kernel/sched/core.c:6510 > #2 0xffffffffb3cf0603 in schedule () at kernel/sched/core.c:6586 > #3 0xffffffffb3308752 in rescuer_thread (__rescuer=0xffff9908001f7480) at > kernel/workqueue.c:2599 > ... > crash> bt > PID: 5 TASK: ffff9908002a0000 CPU: 0 COMMAND: "netns" > #0 [ffffbf750003bea0] schedule at ffffffffb3cf0603 > #1 [ffffbf750003beb0] rescuer_thread at ffffffffb3308752 > ... > ^^^ we can view pid 5's stack trace with no problem > crash> info threads > Id Target Id Frame > * 1 CPU 0 context_switch (rf=0xffffbf750003be58, > next=0xffff9908002d6500, prev=0x0, rq=0xffff990b6fc2fe80) at > kernel/sched/core.c:5201 > ^^^^^^^^^^^^^^^^^^^^^ now its pid > 5's context > > From my view, I think we can keep the current "gdb thread/CPUs = nr_cpus", > or > be more ambitious, leaving "gdb thread/CPUs = 1", in case there are > systems with thousands of cpus, and we only need this in task.c: > > + gdb_refresh_regcache(0); > - for (int i = 0; i < get_cpus_online(); i++) > - gdb_refresh_regcache(i); > > Do you have any thoughts? > > [1]: https://github.com/liutgnu/crash-dev/commits/one-thread > > Thanks, > Tao Liu > > > Thanks, > > Tao Liu > > > > > > > > > > I can think of solving it in two ways: > > > 1. Having a array in crash_target, which tells what CPU in gdb maps to > > > which task in crash, this can be done without requiring changes to > 'info > > > threads', while we do the 'add_thread_silent' > > > 2. Having all tasks in gdb itself, so it's a one-to-one, so `set_cpu` > > > will then refer to the task ID, I will have to explore more on this. > > > > > > Any comments ? > > > > > > Thanks, > > > Aditya Gupta > > > > > > > > > > > Thanks, > > > > --Alexey > > > > > > > > > > > > On Thu, Mar 14, 2024 at 9:22 PM Tao Liu <l...@redhat.com> wrote: > > > > > > > > > Hi Alexey, > > > > > > > > > > On Thu, Mar 14, 2024 at 6:29 PM Alexey Makhalov > > > > > <alexey.makha...@broadcom.com> wrote: > > > > > > > > > > > > Support for GDB debugging of all tasks active and inactive. > > > > > > Before this commit only active tasks were listed by "info > threads" > > > > > > with "CPU #" as a Target Id. > > > > > > > > > > > > "info threads" will now show all tasks, similar to "ps", example: > > > > > > crash> info threads > > > > > > Id Target Id Frame > > > > > > * 1 0 swapper/0 0xffffffffadba19d4 in > default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 2 0 swapper/1 0xffffffffadba19d4 in > default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 3 0 swapper/2 0xffffffffadba19d4 in > default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 4 0 swapper/3 0xffffffffadba19d4 in > default_idle () at > > > > > arch/x86/kernel/process.c:731 > > > > > > 5 0 swapper/4 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf0000f3e88, next=0xffff9ecb04908000, prev=<optimized > out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > ... > > > > > > 730 970325 taskset 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf006a0fd18, next=0xffff9ecb0aec0000, prev=<optimized > out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 731 975217 sleep 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf005743c20, next=0xffff9ecac0692880, prev=<optimized > out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 732 975228 sleep 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf00696fb58, next=0xffff9ecac0690000, prev=<optimized > out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > ... > > > > > > 876 976084 docker 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf0153dbd10, next=0xffff9ecac0645100, prev=<optimized > out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 877 976085 systemd-userwor 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf0153cbc58, next=0xffff9ecac0645100, prev=<optimized > out>, > > > > > rq=<optimized out>) at kernel/sched/core.c:5372 > > > > > > 878 976086 systemd-userwor 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf0153e3c58, next=0xffffffffaec15a40 <init_task>, > > > > > prev=<optimized out>, rq=<optimized out>) at > kernel/sched/core.c:5372 > > > > > > 879 976087 systemd-userwor 0xffffffffadb97292 in > context_switch > > > > > (rf=0xffffbaf0153ebc58, next=0xffffffffaec15a40 <init_task>, > > > > > prev=<optimized out>, rq=<optimized out>) at > kernel/sched/core.c:5372 > > > > > > Where "Target ID" contains "PID COMM" of the task > > > > > > > > > > > > > > > > I see here, we chose a different tech path of viewing arbitrary > tasks. > > > > > You improved the "info threads" as an alternative to crash "ps" > cmd, > > > > > and use "thread X" to switch to task X and view its stacktrace. > > > > > However mine is using crash "ps" cmd first, getting specific tasks > pid > > > > > or task struct, then using crash cmd "set <pid>" or "set > > > > > <task_struct>" to switch to task X and view its stacktrace. > > > > > > > > > > I notice there is one problem of using the "info threads as ps" > tech > > > > > path, it is difficult to handle the live debug, see the comment > below: > > > > > > > > > > > Example of "731 975217 sleep" debugging, real case, trying to > > > > > > figure out why sleep was stuck in uninterruptable sleep. > > > > > > Backtrace using crash: > > > > > > crash> ps | grep 975217 > > > > > > 975217 969797 3 ffff9ecb3956a880 UN 0.0 0 > 0 > > > > > sleep > > > > > > crash> bt 975217 > > > > > > PID: 975217 TASK: ffff9ecb3956a880 CPU: 3 COMMAND: "sleep" > > > > > > #0 [ffffbaf005743ba0] __schedule at ffffffffadb97292 > > > > > > #1 [ffffbaf005743c60] schedule at ffffffffadb982b8 > > > > > > #2 [ffffbaf005743c80] rwbase_write_lock at ffffffffadb9aed7 > > > > > > #3 [ffffbaf005743cc0] down_write at ffffffffadb9b133 > > > > > > #4 [ffffbaf005743cd0] unlink_file_vma at ffffffffad2b0e2e > > > > > > #5 [ffffbaf005743cf8] free_pgtables at ffffffffad2a47b0 > > > > > > #6 [ffffbaf005743d88] exit_mmap at ffffffffad2b3b8d > > > > > > #7 [ffffbaf005743e80] mmput at ffffffffad08c81f > > > > > > #8 [ffffbaf005743e98] do_exit at ffffffffad09636c > > > > > > #9 [ffffbaf005743ef8] do_group_exit at ffffffffad096c78 > > > > > > RIP: 00007f111c70ddf9 RSP: 00007fff451817e8 RFLAGS: > 00000246 > > > > > > RAX: ffffffffffffffda RBX: 00007f111c8089e0 RCX: > 00007f111c70ddf9 > > > > > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: > 0000000000000000 > > > > > > RBP: 0000000000000000 R8: ffffffffffffff80 R9: > 0000000000000000 > > > > > > R10: 00007fff451817b0 R11: 0000000000000246 R12: > 00007f111c8089e0 > > > > > > R13: 00007f111c80e2e0 R14: 0000000000000002 R15: > 00007f111c80e2c8 > > > > > > ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b > > > > > > > > > > > > Backtrace using gdb (pay attention, task must be selected by > thread Id): > > > > > > crash> thread 731 > > > > > > [Switching to thread 731 ( 975217 sleep)] > > > > > > 5372 switch_to(prev, next, prev); > > > > > > crash> gdb bt > > > > > > #0 0xffffffffadb97292 in context_switch (rf=0xffffbaf005743c20, > > > > > next=0xffff9ecac0692880, prev=<optimized out>, rq=<optimized out>) > at > > > > > kernel/sched/core.c:5372 > > > > > > #1 __schedule (sched_mode=sched_mode@entry=0) at > > > > > kernel/sched/core.c:6696 > > > > > > #2 0xffffffffadb982b8 in schedule () at > kernel/sched/core.c:6772 > > > > > > #3 0xffffffffadb9aed7 in rwbase_write_lock (rwb=rwb@entry > =0xffff9ecaf1831430, > > > > > state=state@entry=2) at kernel/locking/rwbase_rt.c:259 > > > > > > #4 0xffffffffadb9b133 in __down_write (sem=sem@entry > =0xffff9ecaf1831430) > > > > > at kernel/locking/rwsem.c:1474 > > > > > > #5 down_write (sem=sem@entry=0xffff9ecaf1831430) at > > > > > kernel/locking/rwsem.c:1574 > > > > > > #6 0xffffffffad2b0e2e in i_mmap_lock_write (mapping=<optimized > out>) > > > > > at ./include/linux/fs.h:466 > > > > > > #7 unlink_file_vma (vma=vma@entry=0xffff9ecadd566090) at > mm/mmap.c:127 > > > > > > #8 0xffffffffad2a47b0 in free_pgtables (tlb=tlb@entry > =0xffffbaf005743dd0, > > > > > mt=mt@entry=0xffff9ecb28e3b180, vma=0xffff9ecadd566090, vma@entry > =0xffff9ecadd566000, > > > > > floor=floor@entry=0, ceiling=ceiling@entry=0) at mm/memory.c:431 > > > > > > #9 0xffffffffad2b3b8d in exit_mmap (mm=mm@entry > =0xffff9ecb28e3b180) > > > > > at mm/mmap.c:3237 > > > > > > #10 0xffffffffad08c81f in __mmput (mm=0xffff9ecb28e3b180) at > > > > > kernel/fork.c:1204 > > > > > > #11 mmput (mm=mm@entry=0xffff9ecb28e3b180) at > kernel/fork.c:1226 > > > > > > #12 0xffffffffad09636c in exit_mm () at kernel/exit.c:563 > > > > > > #13 do_exit (code=code@entry=0) at kernel/exit.c:856 > > > > > > #14 0xffffffffad096c78 in do_group_exit (exit_code=0) at > > > > > kernel/exit.c:1019 > > > > > > #15 0xffffffffad096cf8 in __do_sys_exit_group > (error_code=<optimized > > > > > out>) at kernel/exit.c:1030 > > > > > > #16 __se_sys_exit_group (error_code=<optimized out>) at > > > > > kernel/exit.c:1028 > > > > > > #17 __x64_sys_exit_group (regs=<optimized out>) at > kernel/exit.c:1028 > > > > > > #18 0xffffffffadb8a327 in do_syscall_x64 (nr=<optimized out>, > > > > > regs=0xffffbaf005743f58) at arch/x86/entry/common.c:51 > > > > > > #19 do_syscall_64 (regs=0xffffbaf005743f58, nr=<optimized out>) > at > > > > > arch/x86/entry/common.c:81 > > > > > > #20 0xffffffffadc000dc in entry_SYSCALL_64 () at > > > > > arch/x86/entry/entry_64.S:120 > > > > > > #21 0x00007f111c80e2c8 in ?? () > > > > > > #22 0x0000000000000002 in ?? () > > > > > > #23 0x00007f111c80e2e0 in ?? () > > > > > > #24 0x00007f111c8089e0 in ?? () > > > > > > #25 0x0000000000000000 in ?? () > > > > > > crash> f 3 > > > > > > 259 rwbase_schedule(); > > > > > > crash> p *rwb > > > > > > $1 = { > > > > > > readers = { > > > > > > counter = 1 > > > > > > }, > > > > > > rtmutex = { > > > > > > wait_lock = { > > > > > > raw_lock = { > > > > > > { > > > > > > val = { > > > > > > counter = 0 > > > > > > }, > > > > > > { > > > > > > locked = 0 '\000', > > > > > > pending = 0 '\000' > > > > > > }, > > > > > > { > > > > > > locked_pending = 0, > > > > > > tail = 0 > > > > > > } > > > > > > } > > > > > > } > > > > > > }, > > > > > > waiters = { > > > > > > rb_root = { > > > > > > rb_node = 0xffffbaf006977be0 > > > > > > }, > > > > > > rb_leftmost = 0xffffbaf00696fbe0 > > > > > > }, > > > > > > owner = 0xffff9ecb3956a881 > > > > > > } > > > > > > } > > > > > > > > > > > > Additional changes: > > > > > > 1. Allow gdb "frame" command. > > > > > > 2. Blacklist useless gdb "gcore" command. Use gcore plugin > instead. > > > > > > 3. Move crash_target_init() to later time as crash target > requires a > > > > > list of > > > > > > tasks to be initialized. > > > > > > > > > > > > Known issues and TBD items: > > > > > > 1. "info threads" may bail out first time throwing errors trying > to > > > > > access > > > > > > userspace address during unwind process. Following "info > threads" > > > > > > invokations run without issues. > > > > > > 2. To unwind a stack of inactive task, only modern Linux > versions, which > > > > > use > > > > > > inactive_task_frame, are supported and only x86_64 > architecture. > > > > > > 3. gdb bt unwinder does not stop properly and may show invalid > frames > > > > > (21-25 > > > > > > on example above). Not a regression, existed before. > > > > > > 4. gdb bt unwinder does not work on active tasks in userspace. > Not a > > > > > regression, > > > > > > existed before. > > > > > > 5. Only x86_64 architecture supported. machdep->get_task_reg() > must be > > > > > > implemented for others. Not a regression, existed before. > > > > > > 6. Active tasks registers fetching imlemented only for VMware > dumps, see > > > > > > x86_64_get_task_reg() for more details. Not a regression, > existed > > > > > before. > > > > > > > > > > > > Signed-off-by: Alexey Makhalov <alexey.makha...@broadcom.com> > > > > > > --- > > > > > > crash_target.c | 39 ++++++++++++++++++++--------- > > > > > > defs.h | 10 ++++++-- > > > > > > gdb-10.2.patch | 7 ++---- > > > > > > gdb_interface.c | 63 > ++++++++++++++++++++++++++++++---------------- > > > > > > help.c | 1 + > > > > > > main.c | 1 + > > > > > > task.c | 1 + > > > > > > x86_64.c | 66 > +++++++++++++++++++++++++++++++++++++++++++------ > > > > > > 8 files changed, 140 insertions(+), 48 deletions(-) > > > > > > > > > > > > diff --git a/crash_target.c b/crash_target.c > > > > > > index 4554806..2fdf203 100644 > > > > > > --- a/crash_target.c > > > > > > +++ b/crash_target.c > > > > > > @@ -2,6 +2,8 @@ > > > > > > * crash_target.c > > > > > > * > > > > > > * Copyright (c) 2021 VMware, Inc. > > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term > "Broadcom" > > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > > > * > > > > > > * This program is free software; you can redistribute it > and/or modify > > > > > > * it under the terms of the GNU General Public License as > published by > > > > > > @@ -13,7 +15,7 @@ > > > > > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > > > > * GNU General Public License for more details. > > > > > > * > > > > > > - * Author: Alexey Makhalov <amakha...@vmware.com> > > > > > > + * Author: Alexey Makhalov <alexey.makha...@broadcom.com> > > > > > > */ > > > > > > > > > > > > #include <defs.h> > > > > > > @@ -23,11 +25,11 @@ > > > > > > #include "regcache.h" > > > > > > #include "gdbarch.h" > > > > > > > > > > > > -void crash_target_init (void); > > > > > > - > > > > > > +extern "C" void crash_target_init (void); > > > > > > extern "C" int gdb_readmem_callback(unsigned long, void *, int, > int); > > > > > > -extern "C" int crash_get_nr_cpus(void); > > > > > > -extern "C" int crash_get_cpu_reg (int cpu, int regno, const char > > > > > *regname, > > > > > > +extern "C" int crash_get_nr_tasks(void); > > > > > > +extern "C" void crash_get_task_info(int task_nr, unsigned long > *pid, > > > > > char **comm); > > > > > > +extern "C" int crash_get_task_reg (int task_nr, int regno, > const char > > > > > *regname, > > > > > > int regsize, void *val); > > > > > > > > > > > > > > > > > > @@ -60,7 +62,13 @@ public: > > > > > > bool has_registers () override { return true; } > > > > > > bool thread_alive (ptid_t ptid) override { return true; } > > > > > > std::string pid_to_str (ptid_t ptid) override > > > > > > - { return string_printf ("CPU %ld", ptid.tid ()); } > > > > > > + { > > > > > > + unsigned long pid; > > > > > > + char *comm; > > > > > > + > > > > > > + crash_get_task_info(ptid.tid(), &pid, &comm); > > > > > > + return string_printf ("%7ld %s", pid, comm); > > > > > > + } > > > > > > > > > > > > }; > > > > > > > > > > > > @@ -68,18 +76,25 @@ public: > > > > > > void > > > > > > crash_target::fetch_registers (struct regcache *regcache, int > regno) > > > > > > { > > > > > > + int r; > > > > > > gdb_byte regval[16]; > > > > > > - int cpu = inferior_ptid.tid(); > > > > > > + int task_nr = inferior_ptid.tid(); > > > > > > struct gdbarch *arch = regcache->arch (); > > > > > > > > > > > > - for (int r = 0; r < gdbarch_num_regs (arch); r++) > > > > > > + if (regno >= 0) { > > > > > > + r = regno; > > > > > > + goto onetime; > > > > > > + } > > > > > > + > > > > > > + for (r = 0; regno == -1 && r < gdbarch_num_regs (arch); r++) > > > > > > { > > > > > > +onetime: > > > > > > const char *regname = gdbarch_register_name(arch, r); > > > > > > int regsize = register_size (arch, r); > > > > > > if (regsize > sizeof (regval)) > > > > > > error (_("fatal error: buffer size is not enough to fit > > > > > register value")); > > > > > > > > > > > > - if (crash_get_cpu_reg (cpu, r, regname, regsize, (void > *)®val)) > > > > > > + if (crash_get_task_reg (task_nr, r, regname, regsize, > (void > > > > > *)®val)) > > > > > > regcache->raw_supply (r, regval); > > > > > > else > > > > > > regcache->raw_supply (r, NULL); > > > > > > @@ -107,10 +122,10 @@ crash_target::xfer_partial (enum > target_object > > > > > object, const char *annex, > > > > > > > > > > > > #define CRASH_INFERIOR_PID 1 > > > > > > > > > > > > -void > > > > > > +extern "C" void > > > > > > crash_target_init (void) > > > > > > { > > > > > > - int nr_cpus = crash_get_nr_cpus(); > > > > > > + int nr_tasks = crash_get_nr_tasks(); > > > > > > crash_target *target = new crash_target (); > > > > > > > > > > > > /* Own the target until it is successfully pushed. */ > > > > > > @@ -119,7 +134,7 @@ crash_target_init (void) > > > > > > push_target (std::move (target_holder)); > > > > > > > > > > > > inferior_appeared (current_inferior (), CRASH_INFERIOR_PID); > > > > > > - for (int i = 0; i < nr_cpus; i++) > > > > > > + for (int i = 0; i < nr_tasks; i++) > > > > > > { > > > > > > thread_info *thread = add_thread_silent (target, > > > > > > > ptid_t(CRASH_INFERIOR_PID, 0, > > > > > i)); > > > > > > diff --git a/defs.h b/defs.h > > > > > > index 98650e8..2b3f247 100644 > > > > > > --- a/defs.h > > > > > > +++ b/defs.h > > > > > > @@ -1080,7 +1080,7 @@ struct machdep_table { > > > > > > void (*get_irq_affinity)(int); > > > > > > void (*show_interrupts)(int, ulong *); > > > > > > int (*is_page_ptr)(ulong, physaddr_t *); > > > > > > - int (*get_cpu_reg)(int, int, const char *, int, void *); > > > > > > + int (*get_task_reg)(struct task_context *, int, const > char *, > > > > > int, void *); > > > > > > int (*is_cpu_prstatus_valid)(int cpu); > > > > > > }; > > > > > > > > > > > > @@ -2263,6 +2263,7 @@ struct size_table { /* stash of > > > > > commonly-used sizes */ > > > > > > long pt_regs; > > > > > > long task_struct; > > > > > > long thread_info; > > > > > > + long inactive_task_frame; > > > > > > long softirq_state; > > > > > > long desc_struct; > > > > > > long umode_t; > > > > > > @@ -8001,9 +8002,14 @@ extern int have_full_symbols(void); > > > > > > #define XEN_HYPERVISOR_ARCH > > > > > > #endif > > > > > > > > > > > > +/* > > > > > > + * crash_target.c > > > > > > + */ > > > > > > +extern void crash_target_init (void); > > > > > > + > > > > > > /* > > > > > > * Register numbers must be in sync with > gdb/features/i386/64bit-core.c > > > > > > - * to make crash_target->fetch_registers() ---> > machdep->get_cpu_reg() > > > > > > + * to make crash_target->fetch_registers() ---> > machdep->get_task_reg() > > > > > > * working properly. > > > > > > */ > > > > > > enum x86_64_regnum { > > > > > > diff --git a/gdb-10.2.patch b/gdb-10.2.patch > > > > > > index a7018a2..ecf673d 100644 > > > > > > --- a/gdb-10.2.patch > > > > > > +++ b/gdb-10.2.patch > > > > > > @@ -221,7 +221,7 @@ exit 0 > > > > > > warning (_("\ > > > > > > --- gdb-10.2/gdb/main.c.orig > > > > > > +++ gdb-10.2/gdb/main.c > > > > > > -@@ -392,6 +392,14 @@ start_event_loop () > > > > > > +@@ -392,6 +392,13 @@ start_event_loop () > > > > > > return; > > > > > > } > > > > > > > > > > > > @@ -230,7 +230,6 @@ exit 0 > > > > > > +extern "C" void main_loop(void); > > > > > > +extern "C" unsigned long crash_get_kaslr_offset(void); > > > > > > +extern "C" int console(const char *, ...); > > > > > > -+void crash_target_init (void); > > > > > > +#endif > > > > > > + > > > > > > /* Call command_loop. */ > > > > > > @@ -316,7 +315,7 @@ exit 0 > > > > > > } > > > > > > } > > > > > > > > > > > > -@@ -1242,6 +1274,16 @@ captured_main (void *data) > > > > > > +@@ -1242,6 +1274,14 @@ captured_main (void *data) > > > > > > > > > > > > captured_main_1 (context); > > > > > > > > > > > > @@ -324,8 +323,6 @@ exit 0 > > > > > > + /* Relocate the vmlinux. */ > > > > > > + objfile_rebase (symfile_objfile, crash_get_kaslr_offset()); > > > > > > + > > > > > > -+ crash_target_init(); > > > > > > -+ > > > > > > + /* Back to crash. */ > > > > > > + main_loop(); > > > > > > +#endif > > > > > > diff --git a/gdb_interface.c b/gdb_interface.c > > > > > > index b14319c..03178f5 100644 > > > > > > --- a/gdb_interface.c > > > > > > +++ b/gdb_interface.c > > > > > > @@ -3,6 +3,8 @@ > > > > > > * Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, > Inc. > > > > > > * Copyright (C) 2002-2015,2018-2019 David Anderson > > > > > > * Copyright (C) 2002-2015,2018-2019 Red Hat, Inc. All rights > reserved. > > > > > > + * Copyright (c) 2024 Broadcom. All Rights Reserved. The term > "Broadcom" > > > > > > + * refers to Broadcom Inc. and/or its subsidiaries. > > > > > > * > > > > > > * This program is free software; you can redistribute it > and/or modify > > > > > > * it under the terms of the GNU General Public License as > published by > > > > > > @@ -711,7 +713,7 @@ static char *prohibited_list[] = { > > > > > > "watch", "rwatch", "awatch", "attach", "continue", "c", > "fg", > > > > > "detach", > > > > > > "finish", "handle", "interrupt", "jump", "kill", "next", > "nexti", > > > > > > "signal", "step", "s", "stepi", "target", "until", > "delete", > > > > > > - "clear", "disable", "enable", "condition", "ignore", > "frame", > > > > > "catch", > > > > > > + "clear", "disable", "enable", "condition", "ignore", > "gcore", > > > > > "catch", > > > > > > "tcatch", "return", "file", "exec-file", "core-file", > > > > > "symbol-file", > > > > > > "load", "si", "ni", "shell", "sy", > > > > > > NULL /* must be last */ > > > > > > @@ -877,6 +879,7 @@ gdb_readmem_callback(ulong addr, void *buf, > int len, > > > > > int write) > > > > > > switch (len) > > > > > > { > > > > > > case SIZEOF_8BIT: > > > > > > + fprintf(fp, "%s\n", pc->curcmd); > > > > > > if (STREQ(pc->curcmd, "bt")) { > > > > > > if (readmem(addr, memtype, buf, > SIZEOF_8BIT, > > > > > > "gdb_readmem_callback", readflags)) > > > > > > @@ -1063,34 +1066,52 @@ get_frame_offset(ulong pc) > > > > > > unsigned long crash_get_kaslr_offset(void); > > > > > > unsigned long crash_get_kaslr_offset(void) > > > > > > { > > > > > > - return kt->relocate * -1; > > > > > > + return kt->relocate * -1; > > > > > > } > > > > > > > > > > > > /* Callbacks for crash_target */ > > > > > > -int crash_get_nr_cpus(void); > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > > > +int crash_get_nr_tasks(void); > > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char > **comm); > > > > > > +int crash_get_task_reg (int task_nr, int regno, const char > *regname, > > > > > > int regsize, void *val); > > > > > > > > > > > > -int crash_get_nr_cpus(void) > > > > > > +int crash_get_nr_tasks(void) > > > > > > { > > > > > > - if (SADUMP_DUMPFILE()) > > > > > > - return sadump_get_nr_cpus(); > > > > > > - else if (DISKDUMP_DUMPFILE()) > > > > > > - return diskdump_get_nr_cpus(); > > > > > > - else if (KDUMP_DUMPFILE()) > > > > > > - return kdump_get_nr_cpus(); > > > > > > - else if (VMSS_DUMPFILE()) > > > > > > - return vmware_vmss_get_nr_cpus(); > > > > > > - > > > > > > - /* Just CPU #0 */ > > > > > ()> - return 1; > > > > > > + return RUNNING_TASKS(); > > > > > > } > > > > > > > > > > The crash_get_nr_tasks() will return a fixed number of running > tasks, > > > > > which is the task status when crash is loading. However for live > debug > > > > > mode, there will always be tasks exit and create, so we need to > sync > > > > > the tasks number with the "info threads". I guess it is not an easy > > > > > work because that will involve: > > > > > > > > > > when crash loading: > > > > > for (int i = 0; i < nr_tasks; i++) > > > > > { > > > > > thread_info *thread = add_thread_silent (target, ... > > > > > } > > > > > > > > > > when new task create: > > > > > thread_info *thread = add_thread_silent (target, ... > > > > > > > > > > when task exit: > > > > > delete_thread(...) > > > > > > > > > > We may monitor those task create/exit events and do thread > add/delete > > > > > in a callback function or we just always do all tasks delete/add > when > > > > > "info thread" is invoked. But I guess none of these are as simple > as > > > > > using crash "ps" cmd and "set <pid>", there won't be any problem. > > > > > Because crash "ps" can always show the current tasks status, and > set > > > > > <pid> can switch the task context to it. > > > > > > > > > > What do you think? > > > > > > > > > > Thanks, > > > > > Tao Liu > > > > > > > > > > > > > > > > > -int crash_get_cpu_reg (int cpu, int regno, const char *regname, > > > > > > - int regsize, void *value) > > > > > > +/* Get task information by its index number in TT */ > > > > > > +void crash_get_task_info(int task_nr, unsigned long *pid, char > **comm) > > > > > > { > > > > > > - if (!machdep->get_cpu_reg) > > > > > > - return FALSE; > > > > > > - return machdep->get_cpu_reg(cpu, regno, regname, > regsize, > > > > > value); > > > > > > + int i; > > > > > > + struct task_context *tc; > > > > > > + > > > > > > + tc = FIRST_CONTEXT(); > > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > > > + if (i == task_nr) { > > > > > > + *pid = tc->pid; > > > > > > + *comm = tc->comm; > > > > > > + return; > > > > > > + } > > > > > > + *pid = 0; > > > > > > + *comm = NULL; > > > > > > + return; > > > > > > +} > > > > > > + > > > > > > +int crash_get_task_reg (int task_nr, int regno, const char > *regname, > > > > > > + int regsize, void *value) > > > > > > +{ > > > > > > + int i; > > > > > > + struct task_context *tc; > > > > > > + > > > > > > + if (!machdep->get_task_reg) > > > > > > + return FALSE; > > > > > > + > > > > > > + tc = FIRST_CONTEXT(); > > > > > > + for (i = 0; i < RUNNING_TASKS(); i++, tc++) > > > > > > + if (i == task_nr) { > > > > > > + return machdep->get_task_reg(tc, regno, > regname, > > > > > regsize, value); > > > > > > + } > > > > > > + return FALSE; > > > > > > } > > > > > > > > > > > > diff --git a/help.c b/help.c > > > > > > index a9c4d30..85dbda5 100644 > > > > > > --- a/help.c > > > > > > +++ b/help.c > > > > > > @@ -8520,6 +8520,7 @@ char *version_info[] = { > > > > > > "Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.", > > > > > > "Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, > Inc.", > > > > > > "Copyright (C) 2015, 2021 VMware, Inc.", > > > > > > +"Copyright (C) 2024 Broadcom, Inc.", > > > > > > "This program is free software, covered by the GNU General > Public > > > > > License,", > > > > > > "and you are welcome to change it and/or distribute copies of > it under", > > > > > > "certain conditions. Enter \"help copying\" to see the > conditions.", > > > > > > diff --git a/main.c b/main.c > > > > > > index 0b6b927..13acd2d 100644 > > > > > > --- a/main.c > > > > > > +++ b/main.c > > > > > > @@ -794,6 +794,7 @@ main_loop(void) > > > > > > } else > > > > > > SIGACTION(SIGINT, restart, &pc->sigaction, NULL); > > > > > > > > > > > > + crash_target_init(); > > > > > > /* > > > > > > * Display system statistics and current context. > > > > > > */ > > > > > > diff --git a/task.c b/task.c > > > > > > index ebdb5be..5d26c52 100644 > > > > > > --- a/task.c > > > > > > +++ b/task.c > > > > > > @@ -298,6 +298,7 @@ task_init(void) > > > > > > tt->flags |= THREAD_INFO; > > > > > > } > > > > > > > > > > > > + STRUCT_SIZE_INIT(inactive_task_frame, > "inactive_task_frame"); > > > > > > MEMBER_OFFSET_INIT(task_struct_state, "task_struct", > "state"); > > > > > > MEMBER_SIZE_INIT(task_struct_state, "task_struct", > "state"); > > > > > > if (INVALID_MEMBER(task_struct_state)) { > > > > > > diff --git a/x86_64.c b/x86_64.c > > > > > > index 502817d..b6e36a5 100644 > > > > > > --- a/x86_64.c > > > > > > +++ b/x86_64.c > > > > > > @@ -126,7 +126,7 @@ static int x86_64_get_framesize(struct > bt_info *, > > > > > ulong, ulong, char *); > > > > > > static void x86_64_framesize_debug(struct bt_info *); > > > > > > static void x86_64_get_active_set(void); > > > > > > static int x86_64_get_kvaddr_ranges(struct vaddr_range *); > > > > > > -static int x86_64_get_cpu_reg(int, int, const char *, int, void > *); > > > > > > +static int x86_64_get_task_reg(struct task_context *, int, > const char > > > > > *, int, void *); > > > > > > static int x86_64_verify_paddr(uint64_t); > > > > > > static void GART_init(void); > > > > > > static void x86_64_exception_stacks_init(void); > > > > > > @@ -195,7 +195,7 @@ x86_64_init(int when) > > > > > > machdep->machspec->irq_eframe_link = > UNINITIALIZED; > > > > > > machdep->machspec->irq_stack_gap = UNINITIALIZED; > > > > > > machdep->get_kvaddr_ranges = > x86_64_get_kvaddr_ranges; > > > > > > - machdep->get_cpu_reg = x86_64_get_cpu_reg; > > > > > > + machdep->get_task_reg = x86_64_get_task_reg; > > > > > > if (machdep->cmdline_args[0]) > > > > > > parse_cmdline_args(); > > > > > > if ((string = pc->read_vmcoreinfo("relocate"))) { > > > > > > @@ -891,7 +891,7 @@ x86_64_dump_machdep_table(ulong arg) > > > > > > fprintf(fp, " is_page_ptr: > x86_64_is_page_ptr()\n"); > > > > > > fprintf(fp, " verify_paddr: > x86_64_verify_paddr()\n"); > > > > > > fprintf(fp, " get_kvaddr_ranges: > > > > > x86_64_get_kvaddr_ranges()\n"); > > > > > > - fprintf(fp, " get_cpu_reg: > x86_64_get_cpu_reg()\n"); > > > > > > + fprintf(fp, " get_task_reg: > x86_64_get_task_reg()\n"); > > > > > > fprintf(fp, " init_kernel_pgd: > x86_64_init_kernel_pgd()\n"); > > > > > > fprintf(fp, "clear_machdep_cache: > > > > > x86_64_clear_machdep_cache()\n"); > > > > > > fprintf(fp, " xendump_p2m_create: %s\n", PVOPS_XEN() ? > > > > > > @@ -6398,6 +6398,9 @@ x86_64_ORC_init(void) > > > > > > }; > > > > > > struct ORC_data *orc; > > > > > > > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > > > "inactive_task_frame", "bp"); > > > > > > + MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > > > "inactive_task_frame", "ret_addr"); > > > > > > + > > > > > > if (machdep->flags & FRAMEPOINTER) > > > > > > return; > > > > > > > > > > > > @@ -6455,9 +6458,6 @@ x86_64_ORC_init(void) > > > > > > orc->__stop_orc_unwind = > symbol_value("__stop_orc_unwind"); > > > > > > orc->orc_lookup = symbol_value("orc_lookup"); > > > > > > > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_bp, > > > > > "inactive_task_frame", "bp"); > > > > > > - MEMBER_OFFSET_INIT(inactive_task_frame_ret_addr, > > > > > "inactive_task_frame", "ret_addr"); > > > > > > - > > > > > > orc->has_signal = MEMBER_EXISTS("orc_entry", "signal"); > /* added > > > > > at 6.3 */ > > > > > > orc->has_end = MEMBER_EXISTS("orc_entry", "end"); > /* > > > > > removed at 6.4 */ > > > > > > > > > > > > @@ -9070,14 +9070,64 @@ x86_64_get_kvaddr_ranges(struct > vaddr_range *vrp) > > > > > > } > > > > > > > > > > > > static int > > > > > > -x86_64_get_cpu_reg(int cpu, int regno, const char *name, > > > > > > +x86_64_get_task_reg(struct task_context *tc, int regno, const > char > > > > > *name, > > > > > > int size, void *value) > > > > > > { > > > > > > if (regno >= LAST_REGNUM) > > > > > > return FALSE; > > > > > > > > > > > > + /* > > > > > > + * For inactive task, grab rip, rbp, rbx, r12, r13, r14 > and r15 > > > > > from > > > > > > + * inactive_task_frame (see __switch_to_asm). Other regs > saved on > > > > > > + * regular frame. > > > > > > + */ > > > > > > + if (!is_task_active(tc->task)) { > > > > > > + int frame_size = > STRUCT_SIZE("inactive_task_frame"); > > > > > > + > > > > > > + /* Only modern kernels supported. */ > > > > > > + if (tt->flags & THREAD_INFO && frame_size == 7 * > 8) { > > > > > > + ulong rsp; > > > > > > + int offset = 0; > > > > > > + switch (regno) { > > > > > > + case RSP_REGNUM: > > > > > > + readmem(tc->task + > > > > > OFFSET(task_struct_thread) + > > > > > > + > > > > > OFFSET(thread_struct_rsp), KVADDR, > > > > > > + &rsp, > sizeof(void *), > > > > > > + "thread_struct > rsp", > > > > > FAULT_ON_ERROR); > > > > > > + rsp += frame_size; > > > > > > + memcpy(value, &rsp, > size); > > > > > > + return TRUE; > > > > > > + case RIP_REGNUM: > > > > > > + offset += 8; > > > > > > + case RBP_REGNUM: > > > > > > + offset += 8; > > > > > > + case RBX_REGNUM: > > > > > > + offset += 8; > > > > > > + case R12_REGNUM: > > > > > > + offset += 8; > > > > > > + case R13_REGNUM: > > > > > > + offset += 8; > > > > > > + case R14_REGNUM: > > > > > > + offset += 8; > > > > > > + case R15_REGNUM: > > > > > > + readmem(tc->task + > > > > > OFFSET(task_struct_thread) + > > > > > > + > > > > > OFFSET(thread_struct_rsp), KVADDR, > > > > > > + &rsp, > sizeof(void *), > > > > > > + "thread_struct > rsp", > > > > > FAULT_ON_ERROR); > > > > > > + readmem(rsp + offset, > KVADDR, > > > > > value, sizeof(void *), > > > > > > + > > > > > "inactive_thread_frame saved regs", FAULT_ON_ERROR); > > > > > > + return TRUE; > > > > > > + } > > > > > > + } > > > > > > + /* TBD: older kernels support. */ > > > > > > + return FALSE; > > > > > > + } > > > > > > + > > > > > > + /* > > > > > > + * Task is active, grab CPU's registers > > > > > > + */ > > > > > > if (VMSS_DUMPFILE()) > > > > > > - return vmware_vmss_get_cpu_reg(cpu, regno, > name, size, > > > > > value); > > > > > > + return vmware_vmss_get_cpu_reg(tc->processor, > regno, > > > > > name, size, value); > > > > > > > > > > > > return FALSE; > > > > > > } > > > > > > -- > > > > > > 2.39.0 > > > > > > > > > > > > > > > > > > > > >
-- Crash-utility mailing list -- devel@lists.crash-utility.osci.io To unsubscribe send an email to devel-le...@lists.crash-utility.osci.io https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/ Contribution Guidelines: https://github.com/crash-utility/crash/wiki
