On Thu, 7 May 2020 at 02:21, Nan Xiao <[email protected]> wrote:
> I have a question about kthr:r metric of vmstat command. From manual:
>
> kthr
> Report the number of kernel threads in each of the
> three following states:
>
> r
> the number of kernel threads in run queue
>
> So does "run queue" includes both kernel threads waiting to be serviced
> and those currently being serviced or only waiting to be serviced?
Hi!
I find the best way to start this kind of exploration is to look at the
tool that prints the data or the message that you're interested in
understanding. In this case, that's "vmstat":
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/cmd/stat/vmstat
You can see the "kthr" section of the header here:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/cmd/stat/vmstat/vmstat.c#390
That function, "printhdr()" is called before we print the values:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/cmd/stat/vmstat/vmstat.c#275
You're not specifying the "-p" flag, so the first column of data output
then happens on 317:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/cmd/stat/vmstat/vmstat.c#317
In particular, the first "r" value appears to come from:
static void
dovmstats(struct snapshot *old, struct snapshot *new)
{
...
adjprintf(" %*lu", 1, DELTA(s_sys.ss_sysinfo.runque) / sys_updates);
The DELTA() macro uses the "old" and "new" parameters, each of which has
a "struct sys_snapshot s_sys" member. Then, "s_sys" has a "sysinfo_t
ss_sysinfo" member. The "sysinfo_t" structure comes from the kernel:
typedef struct sysinfo { /* (update freq) update action */
...
uint_t runque; /* (1 sec) += num runnable procs */
We read this value out of a kstat:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/cmd/stat/common/acquire.c#249
int
acquire_sys(struct snapshot *ss, kstat_ctl_t *kc)
{
...
if ((ksp = kstat_lookup(kc, "unix", 0, "sysinfo")) == NULL)
return (errno);
if (kstat_read(kc, ksp, &ss->s_sys.ss_sysinfo) == -1)
return (errno);
This kstat is registered early in the boot of the kernel:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/uts/common/os/kstat_fr.c#522
Note that this is a KSTAT_TYPE_RAW kstat, which allows a struct or other
blob of data with a defined size to be read out by processes through the
kstat interface. In this case, we've set the raw pointer to point to
the global "sysinfo" object:
extern sysinfo_t sysinfo;
...
ksp = kstat_create("unix", 0, "sysinfo", "misc", KSTAT_TYPE_RAW,
sizeof (sysinfo_t), KSTAT_FLAG_VIRTUAL);
if (ksp) {
ksp->ks_data = (void *) &sysinfo;
kstat_install(ksp);
}
There are only a few places in the kernel where we manipulate that
"sysinfo" object, and the most important is in the "clock()" routine
which runs regularly for housekeeping functions:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/uts/common/os/clock.c#862
This is the value that is then read out periodically by kstat(1M) to
display the "kthr:r" value you're after. I leave it as an exercise to
read through the comments and the code further up to see how exactly we
arrive at the "nrunnable" number:
https://code.illumos.org/plugins/gitiles/illumos-gate/+/refs/heads/master/usr/src/uts/common/os/clock.c#457
When reading a large codebase like this, I'd encourage you to use a code
search tool like "cscope" which allows you to locate uses of particular
variables or functions in a somewhat C source aware fashion. That's how
I was able to follow the flow above!
Cheers.
--
Joshua M. Clulow
http://blog.sysmgr.org
------------------------------------------
illumos: illumos-discuss
Permalink:
https://illumos.topicbox.com/groups/discuss/T2d7b5ce8a6fb0f14-Mc7073506f66241b033e0fc8d
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
