Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
The pull request you sent on Fri, 27 Nov 2020 14:47:15 +0100: > git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git > tags/printk-for-5.10-rc6-fixup has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/43d6ecd97c0c69acffc918cc18cdabdfcaa55354 Thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
The pull request you sent on Fri, 16 Oct 2020 17:05:54 +0200: > git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux > tags/printk-for-5.10-fixup has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/8119c4332d253660e0a6b8748fe0749961cfbc97 Thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
On Wed 2020-10-14 16:58:27, Rasmus Villemoes wrote:
> On 14/10/2020 16.16, Geert Uytterhoeven wrote:
> > Hi Petr,
> >
> > On Mon, Oct 12, 2020 at 4:50 PM Petr Mladek wrote:
> >> - Fully lockless ringbuffer implementation, including the support for
> >> continuous lines. It will allow to store and read messages in any
> >> situation wihtout the risk of deadlocks and without the need
> >> of temporary per-CPU buffers.
> >
> > linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old
> > vmlinux.lockless_ringbuffer
> > add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713)
> > Function old new delta
> > _printk_rb_static_infos- 180224 +180224
> > _printk_rb_static_descs- 24576 +24576
> > [...]
> >
> > Seriously?!? Or am I being misled by the tools?
> >
> > linux-m68k-atari_defconfig$ size vmlinux.old vmlinux.lockless_ringbuffer
> >textdata bss dec hex filename
> > 3559108 941716 12 4678596 4763c4 vmlinux.old
> > 3563922 1152496 175276 4891694 4aa42e vmlinux.lockless_ringbuffer
> >
> > Apparently not...
>
> Hm, that's quite a lot. And the only reason the buffers don't live
> entirely in .bss is because a few of their entries have non-zero
> initializers.
>
> Perhaps one could add a .init.text.initialize_static_data section of
> function pointers, with the _DEFINE_PRINTKRB macro growing something like
>
> static void __init __initialize_printkrb_##name(void) { \
> _##name##_descs[_DESCS_COUNT(descbits) - 1] = ...; \
> _##name##_infos[0] = ...; \
> _##name##_infos[_DESCS_COUNT(descbits) - 1] = ...; \
> } \
> static_data_initializer(__initialize_printkrb_##name);
>
> with static_data_initalizer being the obvious yoga for putting a
> function pointer in the .init.text.initialize_static_data section. Then
> very early in start_kernel(), probably first thing, iterate that section
> and call all the functions. But maybe that's not even early enough?
A solution might be to initialize the buffer during the first
printk() call. We could make sure that it is done in
setup_log_buf() at latest. It is called when only one CPU is
running so it should be safe. The only problem might be NMI.
Best Regards,
Petr
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
On Wed 2020-10-14 23:01:30, John Ogness wrote: > On 2020-10-14, Geert Uytterhoeven wrote: > >> - Fully lockless ringbuffer implementation, including the support for > >> continuous lines. It will allow to store and read messages in any > >> situation wihtout the risk of deadlocks and without the need > >> of temporary per-CPU buffers. > > > > linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old > > vmlinux.lockless_ringbuffer > > add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713) > > Function old new delta > > _printk_rb_static_infos- 180224 +180224 > > _printk_rb_static_descs- 24576 +24576 > > 131072 of _printk_rb_static_infos is reserved for dictionary usage. The > rest (49152) is for the record meta data. Previously any dictionary > usage and record meta data was embedded in the message buffer (log_buf, > 65536). > > Since the meta data is now separate, setting CONFIG_LOG_BUF_SHIFT=15 > would provide roughly the same amount of record storage as > CONFIG_LOG_BUF_SHIFT=16 did with vmlinux.old. Then there would be: > > 32768: message buffer > 24576: meta data > 65536: dictionary data > 12288: descriptor data > > Excluding the dictionary data, the total is 65536. > > (With vmlinux.old there is 65536 total with CONFIG_LOG_BUF_SHIFT=16.) > > It is the reserved dictionary data that is hurting us here. > > Should we provide a config option to kill the dictionary data? This might be the best solution. AFAIK, they are currently used only by journalctl. IMHO, most people are not aware of it and it is not a widely used feature. > Should CONFIG_LOG_BUF_SHIFT do an implicit -1 so that the sizes (without > dictionary data) are about the same as before? Beware that this might result in slightly reduced amount of stored messages. The storage of the dictionary meta data is less effective because they are not longer stored together with the messages and we need to guarantee a space for them. I would prefer to keep the current global default. In average, the size of the systems is growing. They produce more messages and have more memory available. Yeah, the space hurts on small systems. We might modify the default for them. Or we could allow to disable the dictionary. Note that this is not the full picture of the printk memory usage. I believe that we will be able to get rid of printk_safe and printk_nmi. It would safe 16kB per-CPU at runtime. If we make the dictionary optional and remove the per-CPU buffers then we might end-up with less memory needs than before. Best Regards, Petr
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
On 2020-10-14, Geert Uytterhoeven wrote: >> - Fully lockless ringbuffer implementation, including the support for >> continuous lines. It will allow to store and read messages in any >> situation wihtout the risk of deadlocks and without the need >> of temporary per-CPU buffers. > > linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old > vmlinux.lockless_ringbuffer > add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713) > Function old new delta > _printk_rb_static_infos- 180224 +180224 > _printk_rb_static_descs- 24576 +24576 131072 of _printk_rb_static_infos is reserved for dictionary usage. The rest (49152) is for the record meta data. Previously any dictionary usage and record meta data was embedded in the message buffer (log_buf, 65536). Since the meta data is now separate, setting CONFIG_LOG_BUF_SHIFT=15 would provide roughly the same amount of record storage as CONFIG_LOG_BUF_SHIFT=16 did with vmlinux.old. Then there would be: 32768: message buffer 24576: meta data 65536: dictionary data 12288: descriptor data Excluding the dictionary data, the total is 65536. (With vmlinux.old there is 65536 total with CONFIG_LOG_BUF_SHIFT=16.) It is the reserved dictionary data that is hurting us here. Should we provide a config option to kill the dictionary data? Should CONFIG_LOG_BUF_SHIFT do an implicit -1 so that the sizes (without dictionary data) are about the same as before? Maybe dictionaries should only exist in the dynamically allocated ringbuffer? John Ogness
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
Hi Rasmus,
On Wed, Oct 14, 2020 at 4:58 PM Rasmus Villemoes
wrote:
> On 14/10/2020 16.16, Geert Uytterhoeven wrote:
> > On Mon, Oct 12, 2020 at 4:50 PM Petr Mladek wrote:
> >> - Fully lockless ringbuffer implementation, including the support for
> >> continuous lines. It will allow to store and read messages in any
> >> situation wihtout the risk of deadlocks and without the need
> >> of temporary per-CPU buffers.
> >
> > linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old
> > vmlinux.lockless_ringbuffer
> > add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713)
> > Function old new delta
> > _printk_rb_static_infos- 180224 +180224
> > _printk_rb_static_descs- 24576 +24576
> > [...]
> >
> > Seriously?!? Or am I being misled by the tools?
> >
> > linux-m68k-atari_defconfig$ size vmlinux.old vmlinux.lockless_ringbuffer
> >textdata bss dec hex filename
> > 3559108 941716 12 4678596 4763c4 vmlinux.old
> > 3563922 1152496 175276 4891694 4aa42e vmlinux.lockless_ringbuffer
> >
> > Apparently not...
>
> Hm, that's quite a lot. And the only reason the buffers don't live
> entirely in .bss is because a few of their entries have non-zero
> initializers.
Even if this would live in BSS, it would still consume 200 KiB of RAM.
Or am I missing something?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
On 14/10/2020 16.16, Geert Uytterhoeven wrote:
> Hi Petr,
>
> On Mon, Oct 12, 2020 at 4:50 PM Petr Mladek wrote:
>> - Fully lockless ringbuffer implementation, including the support for
>> continuous lines. It will allow to store and read messages in any
>> situation wihtout the risk of deadlocks and without the need
>> of temporary per-CPU buffers.
>
> linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old
> vmlinux.lockless_ringbuffer
> add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713)
> Function old new delta
> _printk_rb_static_infos- 180224 +180224
> _printk_rb_static_descs- 24576 +24576
> [...]
>
> Seriously?!? Or am I being misled by the tools?
>
> linux-m68k-atari_defconfig$ size vmlinux.old vmlinux.lockless_ringbuffer
>textdata bss dec hex filename
> 3559108 941716 12 4678596 4763c4 vmlinux.old
> 3563922 1152496 175276 4891694 4aa42e vmlinux.lockless_ringbuffer
>
> Apparently not...
Hm, that's quite a lot. And the only reason the buffers don't live
entirely in .bss is because a few of their entries have non-zero
initializers.
Perhaps one could add a .init.text.initialize_static_data section of
function pointers, with the _DEFINE_PRINTKRB macro growing something like
static void __init __initialize_printkrb_##name(void) { \
_##name##_descs[_DESCS_COUNT(descbits) - 1] = ...; \
_##name##_infos[0] = ...; \
_##name##_infos[_DESCS_COUNT(descbits) - 1] = ...; \
} \
static_data_initializer(__initialize_printkrb_##name);
with static_data_initalizer being the obvious yoga for putting a
function pointer in the .init.text.initialize_static_data section. Then
very early in start_kernel(), probably first thing, iterate that section
and call all the functions. But maybe that's not even early enough?
Rasmus
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
Hi Petr,
On Mon, Oct 12, 2020 at 4:50 PM Petr Mladek wrote:
> - Fully lockless ringbuffer implementation, including the support for
> continuous lines. It will allow to store and read messages in any
> situation wihtout the risk of deadlocks and without the need
> of temporary per-CPU buffers.
linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old
vmlinux.lockless_ringbuffer
add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713)
Function old new delta
_printk_rb_static_infos- 180224 +180224
_printk_rb_static_descs- 24576 +24576
[...]
Seriously?!? Or am I being misled by the tools?
linux-m68k-atari_defconfig$ size vmlinux.old vmlinux.lockless_ringbuffer
textdata bss dec hex filename
3559108 941716 12 4678596 4763c4 vmlinux.old
3563922 1152496 175276 4891694 4aa42e vmlinux.lockless_ringbuffer
Apparently not...
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)
The pull request you sent on Mon, 12 Oct 2020 16:49:16 +0200: > git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux > tags/printk-for-5.10 has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/d594d8f411d47bf7b583ec3474b11fec348c88bb Thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html
