Re: dmesg content lifetime
On Tue, 22 Nov 2022 19:13:18 +0100 Gary Jennejohn wrote: > On Wed, 23 Nov 2022 01:16:55 +0900 > Tomoaki AOKI wrote: > > > On Tue, 22 Nov 2022 09:47:10 -0600 (CST) > > Dan Mack wrote: > > > > > On Tue, 22 Nov 2022, Alexander Kabaev wrote: > > > > > > > On Tue, 22 Nov 2022 09:12:28 -0600 (CST) > > > > Dan Mack wrote: > > > > > > > >> It seems like dmesg content ages out over time. Is there a way to > > > >> leave the contents based on a fixed memory size instead? > > > >> > > > >> Dan > > > >> > > > > I think this is how it works: the kernel message bugger is of fixed > > > > size and kernel and syslog sequences (dmesg -a) share it. The other > > > > syslog users eventually puts enough content in there to displace all of > > > > kernel messages. If the kernel stays quiet, 'dmesg' then returns > > > > nothing, as by default it filters syslog entries that do not KERN > > > > facility out, see sbin/dmesg/dmesg.c. > > > > > > > > -- > > > > Alexander Kabaev > > > > > > > > > > Thank you Alexander, I did not know this. I'll USL (use-the-source-luke) > > > :-) > > > > > > Dan > > > > Increase kern.msgbufsize tunable on /boot/loader.conf if you want dmesg > > to live longer. For example, recommended value by iwlwifi team is > > 1146880. Much larger than default. > > > > Note that this is actually a tunable and can be set only on boot time. > > > > Or look at /var/run/dmesg.boot. It doesn't get overwritten. > > -- > Gary Jennejohn It may be overwritten on reboot. And there are dmesg.[today|yesterday] on /var/log/, too. Rotated daily. -- Tomoaki AOKI
Re: ULE realtime scheduler advice needed
On Tue, 22 Nov 2022 22:38:04 +0100
Hans Petter Selasky wrote:
> On 11/22/22 20:28, mike tancsa wrote:
> > On 11/17/2022 11:47 PM, Hans Petter Selasky wrote:
> >> Hi,
> >>
> >> I'm doing some work with audio and have noticed some problems with the
> >> ULE scheduler. I have a program that generate audio based on
> >> key-presses. When no keys are pressed, the load is near 0%, but as
> >> soon as you start pressing keys, the load goes maybe to 80% of a CPU
> >> core. This program I run with rtprio 8 xxx. The issue I observe or
> >> hear actually, is that it takes too long until the scheduler grasps
> >> that this program needs it's own CPU core and stops time-sharing the
> >> program. When I however use cpuset -l xxx rtprio 8 yyy everything is
> >> good, and the program outputs realtime audio in-time.
> >>
> >> Or is this perhaps a CPU frequency stepping issue?
> >>
> >> Any advice on where to look?
> >>
> > A long shot, but I am curious if by chance you have hwpstate_intel for
> > your cpu frequency driver. If so, does setting
> > dev.hwpstate_intel.0.epp=0 make any difference ?
> >
>
> Yes, I have four of those, set to 50 by default. Let me try.
>
> --HPS
FYI: I habitally run below manually (as root) when I'm on AC powerline.
sysctl -aN | fgrep dev.hwpstate | fgrep epp | while read OID ; do ; \
sysctl ${OID}=0 ; done
--
Tomoaki AOKI
Re: ULE realtime scheduler advice needed
On 11/22/22 20:28, mike tancsa wrote: On 11/17/2022 11:47 PM, Hans Petter Selasky wrote: Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. Or is this perhaps a CPU frequency stepping issue? Any advice on where to look? A long shot, but I am curious if by chance you have hwpstate_intel for your cpu frequency driver. If so, does setting dev.hwpstate_intel.0.epp=0 make any difference ? Yes, I have four of those, set to 50 by default. Let me try. --HPS
Re: dmesg content lifetime
On Tue, 22 Nov 2022, Rodney W. Grimes wrote: On 22 Nov 2022, at 9:34, Dan Mack wrote: It disappears a piece at a time - the oldest entries disappear first. However, it vanishes even when there are only 2-3 lines in it so I didn't think capacity was in play as I expected. So for example I might see a rate-limit entry from someone spamming the system and then it will usually be gone in a couple days and the buffer is completely empty. Similarly if I do something like ifconfig em0 down; ifconfig em0 up ; it's logged but disappears after a day or so. I'm looking to see if this is just a cron job or something clearing it as it might be user-error on my part. Also this is an older system so I'll probably look at it again after I update. I noticed this too, but discovered with ?dmesg -a? that the buffer was full of syslog messages, so dmesg without -a showed nothing. It seems unfortunate that syslog messages logged in the message buffer, at least once syslogd is running. Apparently this happens because they are output to /dev/console. Mike I very much dislike this behavior. I though that the kernel dmesg buffer was for kernel messages only and that I could always count on going there for any kernel messages about a problem that has occurred, expecting to see my boot time output if nothing had happened since boot. Now instead I am almost always greated with an empty buffer :-(. Rod It's been this way for as long as I can remember. Decades probably. Ted Thank you, Dan On Tue, 22 Nov 2022, Warner Losh wrote: On Tue, Nov 22, 2022 at 8:13 AM Dan Mack wrote: It seems like dmesg content ages out over time. Is there a way to leave the contents based on a fixed memory size instead? It already is a fixed memory size. Do you see it all disappear at once, or over time? Warner -- Rod Grimes rgri...@freebsd.org
Re: ULE realtime scheduler advice needed
On 11/17/2022 11:47 PM, Hans Petter Selasky wrote: Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. Or is this perhaps a CPU frequency stepping issue? Any advice on where to look? A long shot, but I am curious if by chance you have hwpstate_intel for your cpu frequency driver. If so, does setting dev.hwpstate_intel.0.epp=0 make any difference ? ---Mike
Re: dmesg content lifetime
> On 22 Nov 2022, at 9:34, Dan Mack wrote: > > > It disappears a piece at a time - the oldest entries disappear first. > > However, it vanishes even when there are only 2-3 lines in it so I didn't > > think capacity was in play as I expected. > > > > So for example I might see a rate-limit entry from someone spamming the > > system and then it will usually be gone in a couple days and the buffer is > > completely empty. Similarly if I do something like ifconfig em0 down; > > ifconfig em0 up ; it's logged but disappears after a day or so. > > > > I'm looking to see if this is just a cron job or something clearing it as > > it might be user-error on my part. Also this is an older system so I'll > > probably look at it again after I update. > > I noticed this too, but discovered with ?dmesg -a? that the buffer was full > of syslog messages, so dmesg without -a showed nothing. > > It seems unfortunate that syslog messages logged in the message buffer, at > least once syslogd is running. Apparently this happens because they are > output to /dev/console. > > Mike I very much dislike this behavior. I though that the kernel dmesg buffer was for kernel messages only and that I could always count on going there for any kernel messages about a problem that has occurred, expecting to see my boot time output if nothing had happened since boot. Now instead I am almost always greated with an empty buffer :-(. Rod > > > Thank you, > > > > Dan > > > > > > On Tue, 22 Nov 2022, Warner Losh wrote: > > > >> On Tue, Nov 22, 2022 at 8:13 AM Dan Mack wrote: > >> > >>> It seems like dmesg content ages out over time. Is there a way to leave > >>> the contents based on a fixed memory size instead? > >>> > >> > >> It already is a fixed memory size. Do you see it all disappear at once, or > >> over time? > >> > >> Warner > >> > > > -- Rod Grimes rgri...@freebsd.org
Re: dmesg content lifetime
On Wed, 23 Nov 2022 01:16:55 +0900 Tomoaki AOKI wrote: > On Tue, 22 Nov 2022 09:47:10 -0600 (CST) > Dan Mack wrote: > > > On Tue, 22 Nov 2022, Alexander Kabaev wrote: > > > > > On Tue, 22 Nov 2022 09:12:28 -0600 (CST) > > > Dan Mack wrote: > > > > > >> It seems like dmesg content ages out over time. Is there a way to > > >> leave the contents based on a fixed memory size instead? > > >> > > >> Dan > > >> > > > I think this is how it works: the kernel message bugger is of fixed > > > size and kernel and syslog sequences (dmesg -a) share it. The other > > > syslog users eventually puts enough content in there to displace all of > > > kernel messages. If the kernel stays quiet, 'dmesg' then returns > > > nothing, as by default it filters syslog entries that do not KERN > > > facility out, see sbin/dmesg/dmesg.c. > > > > > > -- > > > Alexander Kabaev > > > > > > > Thank you Alexander, I did not know this. I'll USL (use-the-source-luke) > > :-) > > > > Dan > > Increase kern.msgbufsize tunable on /boot/loader.conf if you want dmesg > to live longer. For example, recommended value by iwlwifi team is > 1146880. Much larger than default. > > Note that this is actually a tunable and can be set only on boot time. > Or look at /var/run/dmesg.boot. It doesn't get overwritten. -- Gary Jennejohn
Re: dmesg content lifetime
On Tue, Nov 22, 2022 at 8:49 AM Mike Karels wrote: > On 22 Nov 2022, at 9:34, Dan Mack wrote: > > > It disappears a piece at a time - the oldest entries disappear first. > However, it vanishes even when there are only 2-3 lines in it so I didn't > think capacity was in play as I expected. > > > > So for example I might see a rate-limit entry from someone spamming the > system and then it will usually be gone in a couple days and the buffer is > completely empty. Similarly if I do something like ifconfig em0 down; > ifconfig em0 up ; it's logged but disappears after a day or so. > > > > I'm looking to see if this is just a cron job or something clearing it > as it might be user-error on my part. Also this is an older system so > I'll probably look at it again after I update. > > I noticed this too, but discovered with “dmesg -a” that the buffer was full > of syslog messages, so dmesg without -a showed nothing. > > It seems unfortunate that syslog messages logged in the message buffer, at > least once syslogd is running. Apparently this happens because they are > output to /dev/console. > Output to /dev/console that's not via syslogd goes into this buffer for syslogd to harvest and put in log files. IIRC, though, there's also the messages that syslogd sends to /dev/console in this buffer as well, which can be confusing. I'm not sure what a saner policy would be given both of these use cases. Warner Mike > > > Thank you, > > > > Dan > > > > > > On Tue, 22 Nov 2022, Warner Losh wrote: > > > >> On Tue, Nov 22, 2022 at 8:13 AM Dan Mack wrote: > >> > >>> It seems like dmesg content ages out over time. Is there a way to > leave > >>> the contents based on a fixed memory size instead? > >>> > >> > >> It already is a fixed memory size. Do you see it all disappear at once, > or > >> over time? > >> > >> Warner > >> >
Re: dmesg content lifetime
On Tue, 22 Nov 2022 09:47:10 -0600 (CST) Dan Mack wrote: > On Tue, 22 Nov 2022, Alexander Kabaev wrote: > > > On Tue, 22 Nov 2022 09:12:28 -0600 (CST) > > Dan Mack wrote: > > > >> It seems like dmesg content ages out over time. Is there a way to > >> leave the contents based on a fixed memory size instead? > >> > >> Dan > >> > > I think this is how it works: the kernel message bugger is of fixed > > size and kernel and syslog sequences (dmesg -a) share it. The other > > syslog users eventually puts enough content in there to displace all of > > kernel messages. If the kernel stays quiet, 'dmesg' then returns > > nothing, as by default it filters syslog entries that do not KERN > > facility out, see sbin/dmesg/dmesg.c. > > > > -- > > Alexander Kabaev > > > > Thank you Alexander, I did not know this. I'll USL (use-the-source-luke) > :-) > > Dan Increase kern.msgbufsize tunable on /boot/loader.conf if you want dmesg to live longer. For example, recommended value by iwlwifi team is 1146880. Much larger than default. Note that this is actually a tunable and can be set only on boot time. -- Tomoaki AOKI
Re: ULE realtime scheduler advice needed
On Tue, 22 Nov 2022 13:15:33 +0100 Johan Hendriks wrote: > > On 21/11/2022 21:24, Hans Petter Selasky wrote: > > On 11/21/22 20:12, Mark Johnston wrote: > >> There were some bug fixes earlier this year to address problems where > >> high-priority threads were not getting scheduled quickly enough. If > >> you're using an old kernel, they might improve things: > > > > Are any of these fixes merged to stable/13 ? > > > > --HPS > > > It looks like it. > https://freshbsd.org/freebsd/src?q=sched_ule.c Or track PR on Bugzilla via commit logs on main (pointed by markj@) instead of Differential Revision on Phablicator. Phab shouldn't know about branches other than main this case. On Bugzilla, we can see each of them MFC'ed to stable/13 with second commit-hook. -- Tomoaki AOKI
Re: dmesg content lifetime
On 22 Nov 2022, at 9:34, Dan Mack wrote: > It disappears a piece at a time - the oldest entries disappear first. > However, it vanishes even when there are only 2-3 lines in it so I didn't > think capacity was in play as I expected. > > So for example I might see a rate-limit entry from someone spamming the > system and then it will usually be gone in a couple days and the buffer is > completely empty. Similarly if I do something like ifconfig em0 down; > ifconfig em0 up ; it's logged but disappears after a day or so. > > I'm looking to see if this is just a cron job or something clearing it as it > might be user-error on my part. Also this is an older system so I'll > probably look at it again after I update. I noticed this too, but discovered with “dmesg -a” that the buffer was full of syslog messages, so dmesg without -a showed nothing. It seems unfortunate that syslog messages logged in the message buffer, at least once syslogd is running. Apparently this happens because they are output to /dev/console. Mike > Thank you, > > Dan > > > On Tue, 22 Nov 2022, Warner Losh wrote: > >> On Tue, Nov 22, 2022 at 8:13 AM Dan Mack wrote: >> >>> It seems like dmesg content ages out over time. Is there a way to leave >>> the contents based on a fixed memory size instead? >>> >> >> It already is a fixed memory size. Do you see it all disappear at once, or >> over time? >> >> Warner >>
Re: dmesg content lifetime
On Tue, 22 Nov 2022, Alexander Kabaev wrote: On Tue, 22 Nov 2022 09:12:28 -0600 (CST) Dan Mack wrote: It seems like dmesg content ages out over time. Is there a way to leave the contents based on a fixed memory size instead? Dan I think this is how it works: the kernel message bugger is of fixed size and kernel and syslog sequences (dmesg -a) share it. The other syslog users eventually puts enough content in there to displace all of kernel messages. If the kernel stays quiet, 'dmesg' then returns nothing, as by default it filters syslog entries that do not KERN facility out, see sbin/dmesg/dmesg.c. -- Alexander Kabaev Thank you Alexander, I did not know this. I'll USL (use-the-source-luke) :-) Dan
Re: dmesg content lifetime
On Tue, 22 Nov 2022 09:12:28 -0600 (CST) Dan Mack wrote: > It seems like dmesg content ages out over time. Is there a way to > leave the contents based on a fixed memory size instead? > > Dan > I think this is how it works: the kernel message bugger is of fixed size and kernel and syslog sequences (dmesg -a) share it. The other syslog users eventually puts enough content in there to displace all of kernel messages. If the kernel stays quiet, 'dmesg' then returns nothing, as by default it filters syslog entries that do not KERN facility out, see sbin/dmesg/dmesg.c. -- Alexander Kabaev pgpERzqAOoi9r.pgp Description: Цифровая подпись OpenPGP
Re: dmesg content lifetime
It disappears a piece at a time - the oldest entries disappear first. However, it vanishes even when there are only 2-3 lines in it so I didn't think capacity was in play as I expected. So for example I might see a rate-limit entry from someone spamming the system and then it will usually be gone in a couple days and the buffer is completely empty. Similarly if I do something like ifconfig em0 down; ifconfig em0 up ; it's logged but disappears after a day or so. I'm looking to see if this is just a cron job or something clearing it as it might be user-error on my part. Also this is an older system so I'll probably look at it again after I update. Thank you, Dan On Tue, 22 Nov 2022, Warner Losh wrote: On Tue, Nov 22, 2022 at 8:13 AM Dan Mack wrote: It seems like dmesg content ages out over time. Is there a way to leave the contents based on a fixed memory size instead? It already is a fixed memory size. Do you see it all disappear at once, or over time? Warner
Re: dmesg content lifetime
On Tue, Nov 22, 2022 at 8:13 AM Dan Mack wrote: > It seems like dmesg content ages out over time. Is there a way to leave > the contents based on a fixed memory size instead? > It already is a fixed memory size. Do you see it all disappear at once, or over time? Warner
Re: ULE realtime scheduler advice needed
On 21/11/2022 21:24, Hans Petter Selasky wrote: On 11/21/22 20:12, Mark Johnston wrote: There were some bug fixes earlier this year to address problems where high-priority threads were not getting scheduled quickly enough. If you're using an old kernel, they might improve things: Are any of these fixes merged to stable/13 ? --HPS It looks like it. https://freshbsd.org/freebsd/src?q=sched_ule.c
