Understand numa_faults_memory
Hi,
I am on CentOS 7 with kernel package 3.10.0-862.2.3.el7.x86_64.
I was looking at the scheduling info of my application by cat
/proc/[pid]/sched:
---
se.exec_start: 78998944.120048
se.vruntime : 78337609.962134
se.sum_exec_runtime : 78337613.040860
se.nr_migrations :6
nr_switches : 41
nr_voluntary_switches: 31
nr_involuntary_switches : 10
se.load.weight : 1024
policy :0
prio : 120
clock-delta : 13
mm->numa_scan_seq: 925
numa_migrations, 0
numa_faults_memory, 0, 0, 0, 0, 1
numa_faults_memory, 1, 0, 0, 0, 1
numa_faults_memory, 0, 1, 1, 1, 0
numa_faults_memory, 1, 1, 0, 1, 9
And I am trying to understand the numbers after numa_faults_memory.
I dived into the source code:
https://elixir.bootlin.com/linux/v3.18.75/source/kernel/sched/debug.c
I believe line 539 prints the last 4 lines.
Given my machine has 2 NUMA nodes, and the outer loop will loop through the
NUMA nodes and the inner loop is hardcoded to loop twice, the # of lines in
the output match.
So my questions are:
1. What the inner loop is trying to do? Why looping twice?
2. What is the last number in numa_faults_memory (e.g. 9 for last line)?
3. When will this counter be reset? According to the comment at
https://elixir.bootlin.com/linux/v3.18.75/source/include/linux/sched.h#L1577,
"Exponential decaying average of faults on a per-node basis. Scheduling
placement decisions are made based on the these counts. The values remain
static for the duration of a PTE scan", it sounds like numa_faults_memory
is reset or recompute for each PTE scan. What's PTE scan? What does the
"scheduling" refer to? Like scheduling the migration of a chunk of memory
from one NUMA node to another due to NUMA balancing? If yes, does it mean
if I turn off NUMA balancing("echo 0 > /proc/sys/kernel/numa_balancing"),
PTE scan will stop and numa_faults_memory will remain 0 all the time
(assuming there is sufficient memory all the time)?
4. And for my machine task_struct::numa_faults_memory[] would have 4
elements. Then, I am lost what it is tracking. As I thought it is
tracking the number of numa faults per NUMA node. And the number of
elements is always # of NUMA node * 2.
Thanks!
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
What I mean is will it nowadays compile the kernel with 4.19 or do you need to manually hack the inline asm? On Wed, Oct 17, 2018 at 9:11 PM Carter Cheng wrote: > Well I found and old post by John Criswell describing how he did his > dissertation project SVA. He didn't use LTO but llvm-link. So I figure I > would try to do it that way and see if this works since I am not sure how > good the current support for LTO is. Will clang generally work if an > external assembler is used? > > On Wed, Oct 17, 2018 at 9:05 PM wrote: > >> On Wed, 17 Oct 2018 20:02:46 +0800, Carter Cheng said: >> >> > The problem is I have to do something special with the clang options. I >> > have to add an interprocedural link time optimization pass spitting out >> > bitcode files and tying them together using llvm-link. >> >> As I said back on Friday, this is work that's already been done: >> >> > There's no LTO support in the stock 4.19 tree, but Andi Kleen did a >> patchset >> > for 4.15, and there's another patchset to enable LTO when using Clang >> rather >> > than gcc. (I haven't tried either one, don't use on a production >> machine, as >> > the resulting kernel may crash, eat filesystems, and/or turn your dog >> green...) >> >> http://lmgtfy.com/?q=andi+kleen+linux+4.15+lto >> >> What you're probably going to run into is that adding the options >> isn't the hard part of the project. The hard part will be fixing all >> the places where LTO exposes issues in the code, such as this >> (already-fixed) problem: >> >> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1620485.html >> >> >> ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
Well I found and old post by John Criswell describing how he did his dissertation project SVA. He didn't use LTO but llvm-link. So I figure I would try to do it that way and see if this works since I am not sure how good the current support for LTO is. Will clang generally work if an external assembler is used? On Wed, Oct 17, 2018 at 9:05 PM wrote: > On Wed, 17 Oct 2018 20:02:46 +0800, Carter Cheng said: > > > The problem is I have to do something special with the clang options. I > > have to add an interprocedural link time optimization pass spitting out > > bitcode files and tying them together using llvm-link. > > As I said back on Friday, this is work that's already been done: > > > There's no LTO support in the stock 4.19 tree, but Andi Kleen did a > patchset > > for 4.15, and there's another patchset to enable LTO when using Clang > rather > > than gcc. (I haven't tried either one, don't use on a production > machine, as > > the resulting kernel may crash, eat filesystems, and/or turn your dog > green...) > > http://lmgtfy.com/?q=andi+kleen+linux+4.15+lto > > What you're probably going to run into is that adding the options > isn't the hard part of the project. The hard part will be fixing all > the places where LTO exposes issues in the code, such as this > (already-fixed) problem: > > https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1620485.html > > > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
On Wed, 17 Oct 2018 20:02:46 +0800, Carter Cheng said: > The problem is I have to do something special with the clang options. I > have to add an interprocedural link time optimization pass spitting out > bitcode files and tying them together using llvm-link. As I said back on Friday, this is work that's already been done: > There's no LTO support in the stock 4.19 tree, but Andi Kleen did a patchset > for 4.15, and there's another patchset to enable LTO when using Clang rather > than gcc. (I haven't tried either one, don't use on a production machine, as > the resulting kernel may crash, eat filesystems, and/or turn your dog > green...) http://lmgtfy.com/?q=andi+kleen+linux+4.15+lto What you're probably going to run into is that adding the options isn't the hard part of the project. The hard part will be fixing all the places where LTO exposes issues in the code, such as this (already-fixed) problem: https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1620485.html pgpJ4Y1_TyfcJ.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
Thanks! On Wed, Oct 17, 2018 at 8:12 PM Greg KH wrote: > On Wed, Oct 17, 2018 at 08:02:46PM +0800, Carter Cheng wrote: > > The problem is I have to do something special with the clang options. I > > have to add an interprocedural link time optimization pass spitting out > > bitcode files and tying them together using llvm-link. > > Then add those options to the proper place in the main Makefile. There > really is only "one" kernel makefile, look at the code, it's not that > tricky to follow if you understand make. > > good luck! > > greg k-h > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
On Wed, Oct 17, 2018 at 08:02:46PM +0800, Carter Cheng wrote: > The problem is I have to do something special with the clang options. I > have to add an interprocedural link time optimization pass spitting out > bitcode files and tying them together using llvm-link. Then add those options to the proper place in the main Makefile. There really is only "one" kernel makefile, look at the code, it's not that tricky to follow if you understand make. good luck! greg k-h ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
The problem is I have to do something special with the clang options. I have to add an interprocedural link time optimization pass spitting out bitcode files and tying them together using llvm-link. On Wed, Oct 17, 2018 at 7:55 PM Greg KH wrote: > On Wed, Oct 17, 2018 at 04:24:13PM +0800, Carter Cheng wrote: > > Hi, > > > > I am having exploring how much work it would be to do whole kernel > > optimization using clang on the current kernel. My understanding is the > > kernel uses a recursive make file system. I am curious if there are any > > easy ways to get the kernel to build llvm bit code for all files without > > going through and hand editing hundreds of makefiles. > > There is no need to modify more than one makefile. Look at how the > kernel is built, you should be just fine as-is because people use clang > just fine today. > > good luck! > > greg k-h > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: changing parameters in build
On Wed, Oct 17, 2018 at 04:24:13PM +0800, Carter Cheng wrote: > Hi, > > I am having exploring how much work it would be to do whole kernel > optimization using clang on the current kernel. My understanding is the > kernel uses a recursive make file system. I am curious if there are any > easy ways to get the kernel to build llvm bit code for all files without > going through and hand editing hundreds of makefiles. There is no need to modify more than one makefile. Look at how the kernel is built, you should be just fine as-is because people use clang just fine today. good luck! greg k-h ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
changing parameters in build
Hi, I am having exploring how much work it would be to do whole kernel optimization using clang on the current kernel. My understanding is the kernel uses a recursive make file system. I am curious if there are any easy ways to get the kernel to build llvm bit code for all files without going through and hand editing hundreds of makefiles. Thanks in advance, Carter. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
