Re: Passing boot logs to Linux?
Hi, On 03.01.24 13:44, Heinrich Schuchardt wrote: > On 03.01.24 11:11, Csókás Bence wrote: >> Hi, >> >> 2023. 12. 31. 0:54 keltezéssel, Heinrich Schuchardt írta: >>> On 12/20/23 05:11, Simon Glass wrote: +Heinrich Schuchardt On Tue, 19 Dec 2023 at 13:33, Daniel Golle wrote: > > Hi Bence, > > On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: >> Hi! >> >> Is passing the U-Boot boot log to Linux supported yet? We are working >> with a third-party solution, which works, but is a bit hacky, so I was >> wondering if an official solution has been merged yet. >>> >>> U-Boot supports writing log messages to an rsyslog server via broadcast >>> to UDP port 514 (CONFIG_LOG_SYSLOG=y). >>> >>> The device name can be set via environment variable log_hostname. >>> >>> If you have a log server for your devices in the same network segment, >>> you can consolidate all your logs there. >>> >>> A driver writing log messages to a file could be created in U-Boot. >> >> I am aware, however, this is not what we want to do right now. We want >> to pass the logs to the booted up OS on the *same* machine. >> >> I saw that there was an option CONFIG_CONSOLE_RECORD that saves >> everything to a membuff, but I don't know if that can be exported to >> Linux yet. And if not in the tree yet, would such a patch be welcome? >>> >>> It is not enough to export a memory buffer. There must be support in the >>> operating system to read it. >> >> Of course. But it should be trivial to add a driver to Linux and other >> FOSS OSes to support reading these records from memory. > > Please, have a look at Linux' > > Documentation/admin-guide/ramoops.rst > Documentation/devicetree/bindings/reserved-memory/ramoops.yaml > > If U-Boot had a log driver writing to the ramoops buffer we would not > need any change in Linux. > > The current U-Boot drivers are > > common/log_syslog.c > common/log_console.c Yes, that's what I would suggest as well. For reference, barebox supports pstore with ramoops backend to share its dmesg buffer with Linux. You can take a look there for pointers. The code may not directly portable to U-Boot though barebox implements pstore as a file system. The problem of interactive console I/O filling up the log is resolved by having only logged output (pr_info, dev_info and friends) output to pstore and plain printf only prints to enabled serial consoles. Cheers, Ahmad > > Best regards > > Heinrich > >> >>> Linux supports the common platform error record (CPER) format defined in >>> the UEFI specification. See >>> >>> * https://www.dmtf.org/sites/default/files/PMCI_CPEREvent_Proposal_v3.pdf >> >> According to page 16, records can be of the following types: >> * error, recovered >> * previous error >> * simulated error >> >> There is no type that is an operational condition (even though you can >> append informational sections to an error record). So what we could do >> is mark everything as "recovered", but that is misleading, booting is >> not an "error condition". >> >> > * >> https://uefi.org/sites/default/files/resources/Spike%20Yuan-%20Server%20RAS%20and%20UEFI%20CPER_final.pdf >> >> This looks like some promo material, not very useful from an >> implementation standpoint. Plus, this talks about Intel-based cloud >> computing/HPC, not exactly the embedded systems we are targeting. >> >>> Boot errors can be reported via the ACPI BERT table. But that won't help >>> you in OpenWRT which uses device-trees. >> >> Exactly. I would avoid depending on ACPI/UEFI. Having a simple in-memory >> struct can be supported on all platforms, even without ACPI, UEFI, >> writable filesystem, even DT is not needed if you can pass the base >> pointer to the OS via other means (ie. command line). >> >>> Best regards >>> >>> Heinrich >> >> Bence >> > > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
Re: Passing boot logs to Linux?
On 03.01.24 11:11, Csókás Bence wrote: Hi, 2023. 12. 31. 0:54 keltezéssel, Heinrich Schuchardt írta: On 12/20/23 05:11, Simon Glass wrote: +Heinrich Schuchardt On Tue, 19 Dec 2023 at 13:33, Daniel Golle wrote: Hi Bence, On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: Hi! Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. U-Boot supports writing log messages to an rsyslog server via broadcast to UDP port 514 (CONFIG_LOG_SYSLOG=y). The device name can be set via environment variable log_hostname. If you have a log server for your devices in the same network segment, you can consolidate all your logs there. A driver writing log messages to a file could be created in U-Boot. I am aware, however, this is not what we want to do right now. We want to pass the logs to the booted up OS on the *same* machine. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? It is not enough to export a memory buffer. There must be support in the operating system to read it. Of course. But it should be trivial to add a driver to Linux and other FOSS OSes to support reading these records from memory. Please, have a look at Linux' Documentation/admin-guide/ramoops.rst Documentation/devicetree/bindings/reserved-memory/ramoops.yaml If U-Boot had a log driver writing to the ramoops buffer we would not need any change in Linux. The current U-Boot drivers are common/log_syslog.c common/log_console.c Best regards Heinrich Linux supports the common platform error record (CPER) format defined in the UEFI specification. See * https://www.dmtf.org/sites/default/files/PMCI_CPEREvent_Proposal_v3.pdf According to page 16, records can be of the following types: * error, recovered * previous error * simulated error There is no type that is an operational condition (even though you can append informational sections to an error record). So what we could do is mark everything as "recovered", but that is misleading, booting is not an "error condition". > * https://uefi.org/sites/default/files/resources/Spike%20Yuan-%20Server%20RAS%20and%20UEFI%20CPER_final.pdf This looks like some promo material, not very useful from an implementation standpoint. Plus, this talks about Intel-based cloud computing/HPC, not exactly the embedded systems we are targeting. Boot errors can be reported via the ACPI BERT table. But that won't help you in OpenWRT which uses device-trees. Exactly. I would avoid depending on ACPI/UEFI. Having a simple in-memory struct can be supported on all platforms, even without ACPI, UEFI, writable filesystem, even DT is not needed if you can pass the base pointer to the OS via other means (ie. command line). Best regards Heinrich Bence
Re: Passing boot logs to Linux?
Hi, 2023. 12. 31. 0:54 keltezéssel, Heinrich Schuchardt írta: On 12/20/23 05:11, Simon Glass wrote: +Heinrich Schuchardt On Tue, 19 Dec 2023 at 13:33, Daniel Golle wrote: Hi Bence, On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: Hi! Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. U-Boot supports writing log messages to an rsyslog server via broadcast to UDP port 514 (CONFIG_LOG_SYSLOG=y). The device name can be set via environment variable log_hostname. If you have a log server for your devices in the same network segment, you can consolidate all your logs there. A driver writing log messages to a file could be created in U-Boot. I am aware, however, this is not what we want to do right now. We want to pass the logs to the booted up OS on the *same* machine. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? It is not enough to export a memory buffer. There must be support in the operating system to read it. Of course. But it should be trivial to add a driver to Linux and other FOSS OSes to support reading these records from memory. Linux supports the common platform error record (CPER) format defined in the UEFI specification. See * https://www.dmtf.org/sites/default/files/PMCI_CPEREvent_Proposal_v3.pdf According to page 16, records can be of the following types: * error, recovered * previous error * simulated error There is no type that is an operational condition (even though you can append informational sections to an error record). So what we could do is mark everything as "recovered", but that is misleading, booting is not an "error condition". > * https://uefi.org/sites/default/files/resources/Spike%20Yuan-%20Server%20RAS%20and%20UEFI%20CPER_final.pdf This looks like some promo material, not very useful from an implementation standpoint. Plus, this talks about Intel-based cloud computing/HPC, not exactly the embedded systems we are targeting. Boot errors can be reported via the ACPI BERT table. But that won't help you in OpenWRT which uses device-trees. Exactly. I would avoid depending on ACPI/UEFI. Having a simple in-memory struct can be supported on all platforms, even without ACPI, UEFI, writable filesystem, even DT is not needed if you can pass the base pointer to the OS via other means (ie. command line). Best regards Heinrich Bence
Re: Passing boot logs to Linux?
On 2023-12-31 00:54, Heinrich Schuchardt wrote:
On 12/20/23 05:11, Simon Glass wrote:
+Heinrich Schuchardt
On Tue, 19 Dec 2023 at 13:33, Daniel Golle
wrote:
On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote:
Is passing the U-Boot boot log to Linux supported yet? We are
working
with a third-party solution, which works, but is a bit hacky, so I
was
wondering if an official solution has been merged yet.
U-Boot supports writing log messages to an rsyslog server via broadcast
to UDP port 514 (CONFIG_LOG_SYSLOG=y).
The device name can be set via environment variable log_hostname.
If you have a log server for your devices in the same network segment,
you can consolidate all your logs there.
Sure, that's also an option for one's development environment, but
unfortunately not for various standalone embedded devices in the field.
A driver writing log messages to a file could be created in U-Boot.
Please read our earlier discussion about why writing to persistent
storage is desired and why it isn't. Basically, we need to support both
persistent storage and RAM backends.
I saw that there was an option CONFIG_CONSOLE_RECORD that saves
everything to a membuff, but I don't know if that can be exported to
Linux yet. And if not in the tree yet, would such a patch be
welcome?
It is not enough to export a memory buffer. There must be support in
the
operating system to read it.
Linux supports the common platform error record (CPER) format defined
in
the UEFI specification. See
*
https://www.dmtf.org/sites/default/files/PMCI_CPEREvent_Proposal_v3.pdf
*
https://uefi.org/sites/default/files/resources/Spike%20Yuan-%20Server%20RAS%20and%20UEFI%20CPER_final.pdf
Boot errors can be reported via the ACPI BERT table. But that won't
help
you in OpenWRT which uses device-trees.
This is interesting and presents another option to be implemented at
some point. Just as some devices and platforms want to use UEFI, some
simply don't, and having more options available is always good.
To me this sounds very useful and definitely something we'd like to
have in OpenWrt. So I'd volunteer to review and test your patches
once
they are ready.
What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI
sequences during menu or count-down before boot, so we'd have to
either
introduce a dedicated log_printf() call and use that when ever we
want
the output to also become part of the log buffer or we could somehow
filter the recorded console, eliminating all terminal control
sequences.
I have been asking for this for a while. We need some sort of inline
function to enable ANSI output. Perhaps something like this?
#if CONFIG_IS_ENABLED(ANSI)
bool use_ansi(void);
#else
static inline bool use_ansi(void) { return false; }
#endif
bool use_ansi(void)
{
return gd->flags & GD_FLG_ANSI;
}
Re: Passing boot logs to Linux?
On 12/20/23 05:11, Simon Glass wrote:
+Heinrich Schuchardt
On Tue, 19 Dec 2023 at 13:33, Daniel Golle wrote:
Hi Bence,
On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote:
Hi!
Is passing the U-Boot boot log to Linux supported yet? We are working
with a third-party solution, which works, but is a bit hacky, so I was
wondering if an official solution has been merged yet.
U-Boot supports writing log messages to an rsyslog server via broadcast
to UDP port 514 (CONFIG_LOG_SYSLOG=y).
The device name can be set via environment variable log_hostname.
If you have a log server for your devices in the same network segment,
you can consolidate all your logs there.
A driver writing log messages to a file could be created in U-Boot.
I saw that there was an option CONFIG_CONSOLE_RECORD that saves
everything to a membuff, but I don't know if that can be exported to
Linux yet. And if not in the tree yet, would such a patch be welcome?
It is not enough to export a memory buffer. There must be support in the
operating system to read it.
Linux supports the common platform error record (CPER) format defined in
the UEFI specification. See
* https://www.dmtf.org/sites/default/files/PMCI_CPEREvent_Proposal_v3.pdf
*
https://uefi.org/sites/default/files/resources/Spike%20Yuan-%20Server%20RAS%20and%20UEFI%20CPER_final.pdf
Boot errors can be reported via the ACPI BERT table. But that won't help
you in OpenWRT which uses device-trees.
Best regards
Heinrich
To me this sounds very useful and definitely something we'd like to
have in OpenWrt. So I'd volunteer to review and test your patches once
they are ready.
What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI
sequences during menu or count-down before boot, so we'd have to either
introduce a dedicated log_printf() call and use that when ever we want
the output to also become part of the log buffer or we could somehow
filter the recorded console, eliminating all terminal control sequences.
I have been asking for this for a while. We need some sort of inline
function to enable ANSI output. Perhaps something like this?
#if CONFIG_IS_ENABLED(ANSI)
bool use_ansi(void);
#else
static inline bool use_ansi(void) { return false; }
#endif
bool use_ansi(void)
{
return gd->flags & GD_FLG_ANSI;
}
Cheers
Daniel
Regards,
SImon
Re: Passing boot logs to Linux?
On 2023-12-26 23:38, Csókás Bence wrote: On 2023. 12. 26. 19:09, Dragan Simic wrote: On 2023-12-26 10:46, Simon Glass wrote: On Thu, Dec 21, 2023 at 2:23 AM Dragan Simic wrote: On 2023-12-21 02:44, Dragan Simic wrote: > On 2023-12-21 02:37, Dragan Simic wrote: >> On 2023-12-21 02:03, Daniel Golle wrote: >>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: On 2023-12-21 00:27, Csókás Bence wrote: > Not every system has eMMC/uSD, and as you said, these arguments don't > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > router for example. Whereas RAM is quite cheap nowadays. I see, but I also wonder how many such OpenWrt routers are still used these days, and, even more importantly, how many of them are regularly updated and can be expected to actually use this new feature? >>> >>> Avoid flash writes is a very important matter, even on systems with >>> 128 MiB of SPI-NAND flash which is by far the most common setup you >>> find on off-the-shelf plastic routers and access points nowadays. >> >> I agree, writing something to the SPI chips all the time, no matter >> how small the writes are, is a big no-no, which I already clearly >> expressed in one of my earlier posts. >> >>> Especially also as those devices often come without a local console, >>> having U-Boot's output prepended to dmesg on boot would be a very big >>> win. >> >> I was also thinking about that, but I'm not sure it would be accepted >> to the Linux kernel. Maybe we can try getting that accepted later. Maybe, but just _maybe,_ it could be possible to add a new command-line option to dmesg(1) that would display the last recorded console output, fetched from the pstore. That _might_ get accepted to util-linux, while being perfectly fine from the usability standpoint. I'm also willing to work on that, and I already contributed a few dmesg(1) patches to util-linux. Please, don't get me wrong, I still support having both options available, but I'm also wondering about the target demographic. > > > Plus, I don't want the console subsystem to depend on any file/disk > > > operations/drivers. > > > > Well, the console would still work as usual even if logging to disk > > would fail for any reason, which is similar to the serial console > > still > > working if the graphical console fails. Moreover, if the disk fails, > > the system isn't be able to boot, so any RAM-based console logs > > would be > > lost in that case. All this makes the RAM-based logging no more > > resilient to disk failures. > > Correction: if disk *reads* fail, as well as writes, then the system > will not boot. However, typical failure of Flash media is that it > becomes read-only. That's a good point, but having a read-only root filesystem usually also means having a non-operational system that can only have its stored data salvaged. Unless the system is specifically crafted to survive such scenarios, of course. >>> >>> ... which holds true for any decent embedded OS, which at least >>> allows >>> limited remote access and some kind of recovery even in this >>> situation. >> >> Perhaps. I'm more into running general-purpose Linux distributions on >> single-board computers and derived embedded devices, which are on the >> "thick" end of the embedded device spectrum, so to speak. >> > > I still think that using disk-based pstore is a better option. Just > > as > > you don't want to wear out your flash disks with 30-40 KB of data, I > > also don't want to waste 30-40 KB of RAM. > > As I said, you could just unload the log after you're done processing > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > so you could, in your init, process the in-RAM log, then unload it, then > start your other services, thereby reclaiming that RAM. Using pstore should have that unloading already covered, and the already existing systemd service is there to perform the archiving to the primary filesystem, if desired so. It would all need to be tested in detail, of course. >>> >>> pstore/ramoops uses a statically assigned reserved memory region, so >>> in >>> the moment you want to use that feature you loose that amount of RAM >>> (a >>> few kB, so it doesn't really matter on modern systems). >>> As in: there is *no* dynamic allocation. >>> >>> Imho using pstore/ramoops (which is a more or less Linux-specific >>> debugging feature, meant to store one or more timestamped logs of >>> crashes) might not be the most suitable choice. I understand the >>> advantages of using existing infrastructure, but on the other hand >>> we don't need most of the complexity of pstore for the task. >> >>
Re: Passing boot logs to Linux?
On 2023. 12. 26. 19:09, Dragan Simic wrote: > Hello, > > On 2023-12-26 10:46, Simon Glass wrote: >> On Thu, Dec 21, 2023 at 2:23 AM Dragan Simic wrote: >>> On 2023-12-21 02:44, Dragan Simic wrote: >>> > On 2023-12-21 02:37, Dragan Simic wrote: >>> >> On 2023-12-21 02:03, Daniel Golle wrote: >>> >>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: >>> On 2023-12-21 00:27, Csókás Bence wrote: >>> > Not every system has eMMC/uSD, and as you said, these >>> arguments don't >>> > hold for a 4 MB SPI NAND, for example, one you might find in >>> an OpenWrt >>> > router for example. Whereas RAM is quite cheap nowadays. >>> >>> I see, but I also wonder how many such OpenWrt routers are still >>> used these >>> days, and, even more importantly, how many of them are regularly >>> updated and >>> can be expected to actually use this new feature? >>> >>> >>> >>> Avoid flash writes is a very important matter, even on systems with >>> >>> 128 MiB of SPI-NAND flash which is by far the most common setup you >>> >>> find on off-the-shelf plastic routers and access points nowadays. >>> >> >>> >> I agree, writing something to the SPI chips all the time, no matter >>> >> how small the writes are, is a big no-no, which I already clearly >>> >> expressed in one of my earlier posts. >>> >> >>> >>> Especially also as those devices often come without a local console, >>> >>> having U-Boot's output prepended to dmesg on boot would be a very >>> big >>> >>> win. >>> >> >>> >> I was also thinking about that, but I'm not sure it would be accepted >>> >> to the Linux kernel. Maybe we can try getting that accepted later. >>> >>> Maybe, but just _maybe,_ it could be possible to add a new command-line >>> option to dmesg(1) that would display the last recorded console output, >>> fetched from the pstore. That _might_ get accepted to util-linux, while >>> being perfectly fine from the usability standpoint. >>> >>> I'm also willing to work on that, and I already contributed a few >>> dmesg(1) patches to util-linux. >>> >>> Please, don't get me wrong, I still support having both options >>> available, >>> but I'm also wondering about the target demographic. >>> >>> > > > Plus, I don't want the console subsystem to depend on any >>> file/disk >>> > > > operations/drivers. >>> > > >>> > > Well, the console would still work as usual even if logging >>> to disk >>> > > would fail for any reason, which is similar to the serial >>> console >>> > > still >>> > > working if the graphical console fails. Moreover, if the >>> disk fails, >>> > > the system isn't be able to boot, so any RAM-based console logs >>> > > would be >>> > > lost in that case. All this makes the RAM-based logging no >>> more >>> > > resilient to disk failures. >>> > >>> > Correction: if disk *reads* fail, as well as writes, then the >>> system >>> > will not boot. However, typical failure of Flash media is that it >>> > becomes read-only. >>> >>> That's a good point, but having a read-only root filesystem usually >>> also >>> means having a non-operational system that can only have its stored >>> data >>> salvaged. Unless the system is specifically crafted to survive >>> such >>> scenarios, of course. >>> >>> >>> >>> ... which holds true for any decent embedded OS, which at least >>> >>> allows >>> >>> limited remote access and some kind of recovery even in this >>> >>> situation. >>> >> >>> >> Perhaps. I'm more into running general-purpose Linux >>> distributions on >>> >> single-board computers and derived embedded devices, which are on the >>> >> "thick" end of the embedded device spectrum, so to speak. >>> >> >>> > > I still think that using disk-based pstore is a better >>> option. Just >>> > > as >>> > > you don't want to wear out your flash disks with 30-40 KB of >>> data, I >>> > > also don't want to waste 30-40 KB of RAM. >>> > >>> > As I said, you could just unload the log after you're done >>> processing >>> > it. 40 KB RAM is less, than what `sshd` uses, for instance >>> (860k on my >>> > laptop, but it can probably be less, maybe even 10x less, so >>> 80-90k?), >>> > so you could, in your init, process the in-RAM log, then >>> unload it, then >>> > start your other services, thereby reclaiming that RAM. >>> >>> Using pstore should have that unloading already covered, and the >>> already >>> existing systemd service is there to perform the archiving to the >>> primary >>> filesystem, if desired so. It would all need to be tested in >>> detail, of >>> course. >>> >>> >>> >>> pstore/ramoops uses a statically assigned reserved memory region, so >>> >>> in >>> >>> the moment you want to use that feature you loose that amount of RAM >>> >>> (a >>> >>> few kB, so it
Re: Passing boot logs to Linux?
Hello, On 2023-12-26 10:46, Simon Glass wrote: On Thu, Dec 21, 2023 at 2:23 AM Dragan Simic wrote: On 2023-12-21 02:44, Dragan Simic wrote: > On 2023-12-21 02:37, Dragan Simic wrote: >> On 2023-12-21 02:03, Daniel Golle wrote: >>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: On 2023-12-21 00:27, Csókás Bence wrote: > Not every system has eMMC/uSD, and as you said, these arguments don't > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > router for example. Whereas RAM is quite cheap nowadays. I see, but I also wonder how many such OpenWrt routers are still used these days, and, even more importantly, how many of them are regularly updated and can be expected to actually use this new feature? >>> >>> Avoid flash writes is a very important matter, even on systems with >>> 128 MiB of SPI-NAND flash which is by far the most common setup you >>> find on off-the-shelf plastic routers and access points nowadays. >> >> I agree, writing something to the SPI chips all the time, no matter >> how small the writes are, is a big no-no, which I already clearly >> expressed in one of my earlier posts. >> >>> Especially also as those devices often come without a local console, >>> having U-Boot's output prepended to dmesg on boot would be a very big >>> win. >> >> I was also thinking about that, but I'm not sure it would be accepted >> to the Linux kernel. Maybe we can try getting that accepted later. Maybe, but just _maybe,_ it could be possible to add a new command-line option to dmesg(1) that would display the last recorded console output, fetched from the pstore. That _might_ get accepted to util-linux, while being perfectly fine from the usability standpoint. I'm also willing to work on that, and I already contributed a few dmesg(1) patches to util-linux. Please, don't get me wrong, I still support having both options available, but I'm also wondering about the target demographic. > > > Plus, I don't want the console subsystem to depend on any file/disk > > > operations/drivers. > > > > Well, the console would still work as usual even if logging to disk > > would fail for any reason, which is similar to the serial console > > still > > working if the graphical console fails. Moreover, if the disk fails, > > the system isn't be able to boot, so any RAM-based console logs > > would be > > lost in that case. All this makes the RAM-based logging no more > > resilient to disk failures. > > Correction: if disk *reads* fail, as well as writes, then the system > will not boot. However, typical failure of Flash media is that it > becomes read-only. That's a good point, but having a read-only root filesystem usually also means having a non-operational system that can only have its stored data salvaged. Unless the system is specifically crafted to survive such scenarios, of course. >>> >>> ... which holds true for any decent embedded OS, which at least >>> allows >>> limited remote access and some kind of recovery even in this >>> situation. >> >> Perhaps. I'm more into running general-purpose Linux distributions on >> single-board computers and derived embedded devices, which are on the >> "thick" end of the embedded device spectrum, so to speak. >> > > I still think that using disk-based pstore is a better option. Just > > as > > you don't want to wear out your flash disks with 30-40 KB of data, I > > also don't want to waste 30-40 KB of RAM. > > As I said, you could just unload the log after you're done processing > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > so you could, in your init, process the in-RAM log, then unload it, then > start your other services, thereby reclaiming that RAM. Using pstore should have that unloading already covered, and the already existing systemd service is there to perform the archiving to the primary filesystem, if desired so. It would all need to be tested in detail, of course. >>> >>> pstore/ramoops uses a statically assigned reserved memory region, so >>> in >>> the moment you want to use that feature you loose that amount of RAM >>> (a >>> few kB, so it doesn't really matter on modern systems). >>> As in: there is *no* dynamic allocation. >>> >>> Imho using pstore/ramoops (which is a more or less Linux-specific >>> debugging feature, meant to store one or more timestamped logs of >>> crashes) might not be the most suitable choice. I understand the >>> advantages of using existing infrastructure, but on the other hand >>> we don't need most of the complexity of pstore for the task. >> >> Hmm, let me research all that a bit more in the next couple of days, >> please,
Re: Passing boot logs to Linux?
Hi, On Thu, Dec 21, 2023 at 2:23 AM Dragan Simic wrote: > > On 2023-12-21 02:44, Dragan Simic wrote: > > On 2023-12-21 02:37, Dragan Simic wrote: > >> On 2023-12-21 02:03, Daniel Golle wrote: > >>> On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: > On 2023-12-21 00:27, Csókás Bence wrote: > > Not every system has eMMC/uSD, and as you said, these arguments don't > > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > > router for example. Whereas RAM is quite cheap nowadays. > > I see, but I also wonder how many such OpenWrt routers are still > used these > days, and, even more importantly, how many of them are regularly > updated and > can be expected to actually use this new feature? > >>> > >>> Avoid flash writes is a very important matter, even on systems with > >>> 128 MiB of SPI-NAND flash which is by far the most common setup you > >>> find on off-the-shelf plastic routers and access points nowadays. > >> > >> I agree, writing something to the SPI chips all the time, no matter > >> how small the writes are, is a big no-no, which I already clearly > >> expressed in one of my earlier posts. > >> > >>> Especially also as those devices often come without a local console, > >>> having U-Boot's output prepended to dmesg on boot would be a very big > >>> win. > >> > >> I was also thinking about that, but I'm not sure it would be accepted > >> to the Linux kernel. Maybe we can try getting that accepted later. > > Maybe, but just _maybe,_ it could be possible to add a new command-line > option to dmesg(1) that would display the last recorded console output, > fetched from the pstore. That _might_ get accepted to util-linux, while > being perfectly fine from the usability standpoint. > > I'm also willing to work on that, and I already contributed a few > dmesg(1) patches to util-linux. > > Please, don't get me wrong, I still support having both options > available, > but I'm also wondering about the target demographic. > > > > > Plus, I don't want the console subsystem to depend on any file/disk > > > > operations/drivers. > > > > > > Well, the console would still work as usual even if logging to disk > > > would fail for any reason, which is similar to the serial console > > > still > > > working if the graphical console fails. Moreover, if the disk fails, > > > the system isn't be able to boot, so any RAM-based console logs > > > would be > > > lost in that case. All this makes the RAM-based logging no more > > > resilient to disk failures. > > > > Correction: if disk *reads* fail, as well as writes, then the system > > will not boot. However, typical failure of Flash media is that it > > becomes read-only. > > That's a good point, but having a read-only root filesystem usually > also > means having a non-operational system that can only have its stored > data > salvaged. Unless the system is specifically crafted to survive such > scenarios, of course. > >>> > >>> ... which holds true for any decent embedded OS, which at least > >>> allows > >>> limited remote access and some kind of recovery even in this > >>> situation. > >> > >> Perhaps. I'm more into running general-purpose Linux distributions on > >> single-board computers and derived embedded devices, which are on the > >> "thick" end of the embedded device spectrum, so to speak. > >> > > > I still think that using disk-based pstore is a better option. Just > > > as > > > you don't want to wear out your flash disks with 30-40 KB of data, I > > > also don't want to waste 30-40 KB of RAM. > > > > As I said, you could just unload the log after you're done processing > > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > > so you could, in your init, process the in-RAM log, then unload it, > > then > > start your other services, thereby reclaiming that RAM. > > Using pstore should have that unloading already covered, and the > already > existing systemd service is there to perform the archiving to the > primary > filesystem, if desired so. It would all need to be tested in > detail, of > course. > >>> > >>> pstore/ramoops uses a statically assigned reserved memory region, so > >>> in > >>> the moment you want to use that feature you loose that amount of RAM > >>> (a > >>> few kB, so it doesn't really matter on modern systems). > >>> As in: there is *no* dynamic allocation. > >>> > >>> Imho using pstore/ramoops (which is a more or less Linux-specific > >>> debugging feature, meant to store one or more timestamped logs of > >>> crashes) might not be the most suitable choice. I understand the > >>> advantages of using existing infrastructure, but
Re: Passing boot logs to Linux?
On 2023-12-21 02:44, Dragan Simic wrote: On 2023-12-21 02:37, Dragan Simic wrote: On 2023-12-21 02:03, Daniel Golle wrote: On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: On 2023-12-21 00:27, Csókás Bence wrote: > Not every system has eMMC/uSD, and as you said, these arguments don't > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > router for example. Whereas RAM is quite cheap nowadays. I see, but I also wonder how many such OpenWrt routers are still used these days, and, even more importantly, how many of them are regularly updated and can be expected to actually use this new feature? Avoid flash writes is a very important matter, even on systems with 128 MiB of SPI-NAND flash which is by far the most common setup you find on off-the-shelf plastic routers and access points nowadays. I agree, writing something to the SPI chips all the time, no matter how small the writes are, is a big no-no, which I already clearly expressed in one of my earlier posts. Especially also as those devices often come without a local console, having U-Boot's output prepended to dmesg on boot would be a very big win. I was also thinking about that, but I'm not sure it would be accepted to the Linux kernel. Maybe we can try getting that accepted later. Maybe, but just _maybe,_ it could be possible to add a new command-line option to dmesg(1) that would display the last recorded console output, fetched from the pstore. That _might_ get accepted to util-linux, while being perfectly fine from the usability standpoint. I'm also willing to work on that, and I already contributed a few dmesg(1) patches to util-linux. Please, don't get me wrong, I still support having both options available, but I'm also wondering about the target demographic. > > > Plus, I don't want the console subsystem to depend on any file/disk > > > operations/drivers. > > > > Well, the console would still work as usual even if logging to disk > > would fail for any reason, which is similar to the serial console > > still > > working if the graphical console fails. Moreover, if the disk fails, > > the system isn't be able to boot, so any RAM-based console logs > > would be > > lost in that case. All this makes the RAM-based logging no more > > resilient to disk failures. > > Correction: if disk *reads* fail, as well as writes, then the system > will not boot. However, typical failure of Flash media is that it > becomes read-only. That's a good point, but having a read-only root filesystem usually also means having a non-operational system that can only have its stored data salvaged. Unless the system is specifically crafted to survive such scenarios, of course. ... which holds true for any decent embedded OS, which at least allows limited remote access and some kind of recovery even in this situation. Perhaps. I'm more into running general-purpose Linux distributions on single-board computers and derived embedded devices, which are on the "thick" end of the embedded device spectrum, so to speak. > > I still think that using disk-based pstore is a better option. Just > > as > > you don't want to wear out your flash disks with 30-40 KB of data, I > > also don't want to waste 30-40 KB of RAM. > > As I said, you could just unload the log after you're done processing > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > so you could, in your init, process the in-RAM log, then unload it, then > start your other services, thereby reclaiming that RAM. Using pstore should have that unloading already covered, and the already existing systemd service is there to perform the archiving to the primary filesystem, if desired so. It would all need to be tested in detail, of course. pstore/ramoops uses a statically assigned reserved memory region, so in the moment you want to use that feature you loose that amount of RAM (a few kB, so it doesn't really matter on modern systems). As in: there is *no* dynamic allocation. Imho using pstore/ramoops (which is a more or less Linux-specific debugging feature, meant to store one or more timestamped logs of crashes) might not be the most suitable choice. I understand the advantages of using existing infrastructure, but on the other hand we don't need most of the complexity of pstore for the task. Hmm, let me research all that a bit more in the next couple of days, please, and I'll come back with a detailed insight. What I'd like to see is having a couple of log lines from U-Boot prepended to Linux' dmesg buffer, and for that we anyway will have to come up with a way to hand over that buffer. Another reserved-memory region would be one way, embedding the buffer as a blob into the /chosen/ section of the device tree would be another way. As I already wrote above, it would be rather neat, but I'm afraid it wouldn't be accepted upstream.
Re: Passing boot logs to Linux?
On 2023-12-21 02:37, Dragan Simic wrote: On 2023-12-21 02:03, Daniel Golle wrote: On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: On 2023-12-21 00:27, Csókás Bence wrote: > Not every system has eMMC/uSD, and as you said, these arguments don't > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > router for example. Whereas RAM is quite cheap nowadays. I see, but I also wonder how many such OpenWrt routers are still used these days, and, even more importantly, how many of them are regularly updated and can be expected to actually use this new feature? Avoid flash writes is a very important matter, even on systems with 128 MiB of SPI-NAND flash which is by far the most common setup you find on off-the-shelf plastic routers and access points nowadays. I agree, writing something to the SPI chips all the time, no matter how small the writes are, is a big no-no, which I already clearly expressed in one of my earlier posts. Especially also as those devices often come without a local console, having U-Boot's output prepended to dmesg on boot would be a very big win. I was also thinking about that, but I'm not sure it would be accepted to the Linux kernel. Maybe we can try getting that accepted later. Please, don't get me wrong, I still support having both options available, but I'm also wondering about the target demographic. > > > Plus, I don't want the console subsystem to depend on any file/disk > > > operations/drivers. > > > > Well, the console would still work as usual even if logging to disk > > would fail for any reason, which is similar to the serial console > > still > > working if the graphical console fails. Moreover, if the disk fails, > > the system isn't be able to boot, so any RAM-based console logs > > would be > > lost in that case. All this makes the RAM-based logging no more > > resilient to disk failures. > > Correction: if disk *reads* fail, as well as writes, then the system > will not boot. However, typical failure of Flash media is that it > becomes read-only. That's a good point, but having a read-only root filesystem usually also means having a non-operational system that can only have its stored data salvaged. Unless the system is specifically crafted to survive such scenarios, of course. ... which holds true for any decent embedded OS, which at least allows limited remote access and some kind of recovery even in this situation. Perhaps. I'm more into running general-purpose Linux distributions on single-board computers and derived embedded devices, which are on the "thick" end of the embedded device spectrum, so to speak. > > I still think that using disk-based pstore is a better option. Just > > as > > you don't want to wear out your flash disks with 30-40 KB of data, I > > also don't want to waste 30-40 KB of RAM. > > As I said, you could just unload the log after you're done processing > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > so you could, in your init, process the in-RAM log, then unload it, then > start your other services, thereby reclaiming that RAM. Using pstore should have that unloading already covered, and the already existing systemd service is there to perform the archiving to the primary filesystem, if desired so. It would all need to be tested in detail, of course. pstore/ramoops uses a statically assigned reserved memory region, so in the moment you want to use that feature you loose that amount of RAM (a few kB, so it doesn't really matter on modern systems). As in: there is *no* dynamic allocation. Imho using pstore/ramoops (which is a more or less Linux-specific debugging feature, meant to store one or more timestamped logs of crashes) might not be the most suitable choice. I understand the advantages of using existing infrastructure, but on the other hand we don't need most of the complexity of pstore for the task. Hmm, let me research all that a bit more in the next couple of days, please, and I'll come back with a detailed insight. What I'd like to see is having a couple of log lines from U-Boot prepended to Linux' dmesg buffer, and for that we anyway will have to come up with a way to hand over that buffer. Another reserved-memory region would be one way, embedding the buffer as a blob into the /chosen/ section of the device tree would be another way. As I already wrote above, it would be rather neat, but I'm afraid it wouldn't be accepted upstream. Sorry, I forgot this... As I already explained in one of my earlier posts, not all devices and applications would benefit from having only "in-flight" console data available, i.e. made available through a reserved region of the RAM. Some devices actually need to have the recorded console outputs stored on their eMMC chips or microSD cards, for post-mortem analysis of the boot issues. That's why we actually need
Re: Passing boot logs to Linux?
On 2023-12-21 02:03, Daniel Golle wrote: On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: On 2023-12-21 00:27, Csókás Bence wrote: > Not every system has eMMC/uSD, and as you said, these arguments don't > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > router for example. Whereas RAM is quite cheap nowadays. I see, but I also wonder how many such OpenWrt routers are still used these days, and, even more importantly, how many of them are regularly updated and can be expected to actually use this new feature? Avoid flash writes is a very important matter, even on systems with 128 MiB of SPI-NAND flash which is by far the most common setup you find on off-the-shelf plastic routers and access points nowadays. I agree, writing something to the SPI chips all the time, no matter how small the writes are, is a big no-no, which I already clearly expressed in one of my earlier posts. Especially also as those devices often come without a local console, having U-Boot's output prepended to dmesg on boot would be a very big win. I was also thinking about that, but I'm not sure it would be accepted to the Linux kernel. Maybe we can try getting that accepted later. Please, don't get me wrong, I still support having both options available, but I'm also wondering about the target demographic. > > > Plus, I don't want the console subsystem to depend on any file/disk > > > operations/drivers. > > > > Well, the console would still work as usual even if logging to disk > > would fail for any reason, which is similar to the serial console > > still > > working if the graphical console fails. Moreover, if the disk fails, > > the system isn't be able to boot, so any RAM-based console logs > > would be > > lost in that case. All this makes the RAM-based logging no more > > resilient to disk failures. > > Correction: if disk *reads* fail, as well as writes, then the system > will not boot. However, typical failure of Flash media is that it > becomes read-only. That's a good point, but having a read-only root filesystem usually also means having a non-operational system that can only have its stored data salvaged. Unless the system is specifically crafted to survive such scenarios, of course. ... which holds true for any decent embedded OS, which at least allows limited remote access and some kind of recovery even in this situation. Perhaps. I'm more into running general-purpose Linux distributions on single-board computers and derived embedded devices, which are on the "thick" end of the embedded device spectrum, so to speak. > > I still think that using disk-based pstore is a better option. Just > > as > > you don't want to wear out your flash disks with 30-40 KB of data, I > > also don't want to waste 30-40 KB of RAM. > > As I said, you could just unload the log after you're done processing > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > so you could, in your init, process the in-RAM log, then unload it, then > start your other services, thereby reclaiming that RAM. Using pstore should have that unloading already covered, and the already existing systemd service is there to perform the archiving to the primary filesystem, if desired so. It would all need to be tested in detail, of course. pstore/ramoops uses a statically assigned reserved memory region, so in the moment you want to use that feature you loose that amount of RAM (a few kB, so it doesn't really matter on modern systems). As in: there is *no* dynamic allocation. Imho using pstore/ramoops (which is a more or less Linux-specific debugging feature, meant to store one or more timestamped logs of crashes) might not be the most suitable choice. I understand the advantages of using existing infrastructure, but on the other hand we don't need most of the complexity of pstore for the task. Hmm, let me research all that a bit more in the next couple of days, please, and I'll come back with a detailed insight. What I'd like to see is having a couple of log lines from U-Boot prepended to Linux' dmesg buffer, and for that we anyway will have to come up with a way to hand over that buffer. Another reserved-memory region would be one way, embedding the buffer as a blob into the /chosen/ section of the device tree would be another way. As I already wrote above, it would be rather neat, but I'm afraid it wouldn't be accepted upstream.
Re: Passing boot logs to Linux?
On Thu, Dec 21, 2023 at 12:55:20AM +0100, Dragan Simic wrote: > On 2023-12-21 00:27, Csókás Bence wrote: > > Not every system has eMMC/uSD, and as you said, these arguments don't > > hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt > > router for example. Whereas RAM is quite cheap nowadays. > > I see, but I also wonder how many such OpenWrt routers are still used these > days, and, even more importantly, how many of them are regularly updated and > can be expected to actually use this new feature? Avoid flash writes is a very important matter, even on systems with 128 MiB of SPI-NAND flash which is by far the most common setup you find on off-the-shelf plastic routers and access points nowadays. Especially also as those devices often come without a local console, having U-Boot's output prepended to dmesg on boot would be a very big win. > > Please, don't get me wrong, I still support having both options available, > but I'm also wondering about the target demographic. > > > > > Plus, I don't want the console subsystem to depend on any file/disk > > > > operations/drivers. > > > > > > Well, the console would still work as usual even if logging to disk > > > would fail for any reason, which is similar to the serial console > > > still > > > working if the graphical console fails. Moreover, if the disk fails, > > > the system isn't be able to boot, so any RAM-based console logs > > > would be > > > lost in that case. All this makes the RAM-based logging no more > > > resilient to disk failures. > > > > Correction: if disk *reads* fail, as well as writes, then the system > > will not boot. However, typical failure of Flash media is that it > > becomes read-only. > > That's a good point, but having a read-only root filesystem usually also > means having a non-operational system that can only have its stored data > salvaged. Unless the system is specifically crafted to survive such > scenarios, of course. ... which holds true for any decent embedded OS, which at least allows limited remote access and some kind of recovery even in this situation. > > > > I still think that using disk-based pstore is a better option. Just > > > as > > > you don't want to wear out your flash disks with 30-40 KB of data, I > > > also don't want to waste 30-40 KB of RAM. > > > > As I said, you could just unload the log after you're done processing > > it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my > > laptop, but it can probably be less, maybe even 10x less, so 80-90k?), > > so you could, in your init, process the in-RAM log, then unload it, then > > start your other services, thereby reclaiming that RAM. > > Using pstore should have that unloading already covered, and the already > existing systemd service is there to perform the archiving to the primary > filesystem, if desired so. It would all need to be tested in detail, of > course. pstore/ramoops uses a statically assigned reserved memory region, so in the moment you want to use that feature you loose that amount of RAM (a few kB, so it doesn't really matter on modern systems). As in: there is *no* dynamic allocation. Imho using pstore/ramoops (which is a more or less Linux-specific debugging feature, meant to store one or more timestamped logs of crashes) might not be the most suitable choice. I understand the advantages of using existing infrastructure, but on the other hand we don't need most of the complexity of pstore for the task. What I'd like to see is having a couple of log lines from U-Boot prepended to Linux' dmesg buffer, and for that we anyway will have to come up with a way to hand over that buffer. Another reserved-memory region would be one way, embedding the buffer as a blob into the /chosen/ section of the device tree would be another way.
Re: Passing boot logs to Linux?
On 2023-12-21 00:27, Csókás Bence wrote: On 2023. 12. 20. 20:58, Dragan Simic wrote: On 2023-12-20 20:24, Csókás Bence wrote: On 2023. 12. 20. 9:29, Dragan Simic wrote: On 2023-12-20 08:52, Csókás Bence wrote: That's what I read as well. Is there support for U-boot to write and Linux to read PStores? No and yes, but U-Boot can already read pstore. Please see doc/usage/cmd/pstore.rst for the U-Boot part, and Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. Irrelevant, as we only want to write out the console log to U-Boot, and not the other way around (that's for collecting panic logs, which is already implemented). Quite frankly, this is a bit confusing to me. Why should it be irrelevant, if we want to store the U-Boot console logs into the pstore, and read them later in Linux? That's exactly what you asked about, and it was a good question. Because we want to *write* PStore from U-Boot. The read being already implemented is not very useful, we could only re-read an already-written console log (which is not helpful for us). Well, it all depends on one's viewpoint. See, even if we had no read support already implemented in U-Boot and wanted to just write the data, we'd almost surely have to implement reading as well, so the written stuff can be inspected in the U-Boot console. Another benefit of using pstore would be no permanently wasted RAM for the recorded console contents. Also, having the data recorded to a storage device also goes along with providing permanent records. I'm positively *not gonna save boot logs to disk*, as most embedded systems have Flash-based storage media; writing to them on every boot would be devastating. Writing 30-40 KB of data to an even remotely modern flash-based storage media on each boot is pretty much nothing. OTOH, writing 30-40 KB of data to an SPI chip on each boot would be rather bad, but we aren't going to do that for sure. To put things into perspective, I'm writing this on a Pinebook Pro with an eMMC chip, and it has stored many GBs of data over a few years. Even high-endurance microSD cards are rather reliable in that regard. Not every system has eMMC/uSD, and as you said, these arguments don't hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt router for example. Whereas RAM is quite cheap nowadays. I see, but I also wonder how many such OpenWrt routers are still used these days, and, even more importantly, how many of them are regularly updated and can be expected to actually use this new feature? Please, don't get me wrong, I still support having both options available, but I'm also wondering about the target demographic. Plus, I don't want the console subsystem to depend on any file/disk operations/drivers. Well, the console would still work as usual even if logging to disk would fail for any reason, which is similar to the serial console still working if the graphical console fails. Moreover, if the disk fails, the system isn't be able to boot, so any RAM-based console logs would be lost in that case. All this makes the RAM-based logging no more resilient to disk failures. Correction: if disk *reads* fail, as well as writes, then the system will not boot. However, typical failure of Flash media is that it becomes read-only. That's a good point, but having a read-only root filesystem usually also means having a non-operational system that can only have its stored data salvaged. Unless the system is specifically crafted to survive such scenarios, of course. I still think that using disk-based pstore is a better option. Just as you don't want to wear out your flash disks with 30-40 KB of data, I also don't want to waste 30-40 KB of RAM. As I said, you could just unload the log after you're done processing it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my laptop, but it can probably be less, maybe even 10x less, so 80-90k?), so you could, in your init, process the in-RAM log, then unload it, then start your other services, thereby reclaiming that RAM. Using pstore should have that unloading already covered, and the already existing systemd service is there to perform the archiving to the primary filesystem, if desired so. It would all need to be tested in detail, of course. Also, embedded devices such as the PinePhone would benefit additionally from using disk-based logging, because people often can't do anything in the field if booting fails for some reason, so RAM-based logs would end up being lost, preventing any post-mortem debugging later. Post-mortem analysis could be interesting, true. Yes, and it's often the only way to get to the bottom of the underlying issues, which are sometimes intermittent and can't be reproduced easily later. Perhaps we can actually offer both options, i.e. both logging to RAM and to disk, so everyone can choose what they prefer. I'd also be fine with working on
Re: Passing boot logs to Linux?
On 2023. 12. 20. 20:58, Dragan Simic wrote: > On 2023-12-20 20:24, Csókás Bence wrote: >> On 2023. 12. 20. 9:29, Dragan Simic wrote: >>> On 2023-12-20 08:52, Csókás Bence wrote: That's what I read as well. Is there support for U-boot to write and Linux to read PStores? >>> >>> No and yes, but U-Boot can already read pstore. Please see >>> doc/usage/cmd/pstore.rst for the U-Boot part, and >>> Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. >> >> Irrelevant, as we only want to write out the console log to U-Boot, and >> not the other way around (that's for collecting panic logs, which is >> already implemented). > > Quite frankly, this is a bit confusing to me. Why should it be > irrelevant, if we want to store the U-Boot console logs into the pstore, > and read them later in Linux? That's exactly what you asked about, and > it was a good question. Because we want to *write* PStore from U-Boot. The read being already implemented is not very useful, we could only re-read an already-written console log (which is not helpful for us). >>> Another benefit of using pstore would be no permanently wasted RAM for >>> the recorded console contents. Also, having the data recorded to a >>> storage device also goes along with providing permanent records. >> >> I'm positively *not gonna save boot logs to disk*, as most embedded >> systems have Flash-based storage media; writing to them on every boot >> would be devastating. > > Writing 30-40 KB of data to an even remotely modern flash-based storage > media on each boot is pretty much nothing. OTOH, writing 30-40 KB of > data to an SPI chip on each boot would be rather bad, but we aren't > going to do that for sure. > > To put things into perspective, I'm writing this on a Pinebook Pro with > an eMMC chip, and it has stored many GBs of data over a few years. Even > high-endurance microSD cards are rather reliable in that regard. Not every system has eMMC/uSD, and as you said, these arguments don't hold for a 4 MB SPI NAND, for example, one you might find in an OpenWrt router for example. Whereas RAM is quite cheap nowadays. >> Plus, I don't want the console subsystem to depend on any file/disk >> operations/drivers. > > Well, the console would still work as usual even if logging to disk > would fail for any reason, which is similar to the serial console still > working if the graphical console fails. Moreover, if the disk fails, > the system isn't be able to boot, so any RAM-based console logs would be > lost in that case. All this makes the RAM-based logging no more > resilient to disk failures. Correction: if disk *reads* fail, as well as writes, then the system will not boot. However, typical failure of Flash media is that it becomes read-only. > I still think that using disk-based pstore is a better option. Just as > you don't want to wear out your flash disks with 30-40 KB of data, I > also don't want to waste 30-40 KB of RAM. As I said, you could just unload the log after you're done processing it. 40 KB RAM is less, than what `sshd` uses, for instance (860k on my laptop, but it can probably be less, maybe even 10x less, so 80-90k?), so you could, in your init, process the in-RAM log, then unload it, then start your other services, thereby reclaiming that RAM. > Also, embedded devices such > as the PinePhone would benefit additionally from using disk-based > logging, because people often can't do anything in the field if booting > fails for some reason, so RAM-based logs would end up being lost, > preventing any post-mortem debugging later. Post-mortem analysis could be interesting, true. > Perhaps we can actually offer both options, i.e. both logging to RAM and > to disk, so everyone can choose what they prefer. I'd also be fine with > working on such a dual approach, but it would obviously be more complex. Perhaps that is the best option. But, as Sean has pointed out, U-Boot doesn't support disk-backed PStore anyways... So then, should we implement RAM-based PStore writing for the time being, and then maybe expand it later? Bence
Re: Passing boot logs to Linux?
On 2023. 12. 20. 9:29, Dragan Simic wrote: > On 2023-12-20 08:52, Csókás Bence wrote: >> On 2023. 12. 20. 8:09, Dragan Simic wrote: >>> On 2023-12-20 07:49, Csókás Bence wrote: I don't think that would be a huge problem, Linux userspace can filter ANSI control codes if it wants to. For now, I'd like a byte-for-byte copy of the console, as-is, presented to Linux. >>> >>> But wouldn't recording the control sequences as-is actually be pretty >>> much useless? For example, some user can spend a few minutes >>> scrolling through the boot menu, which would produce a fair amount of >>> nearly useless recorded data. >>> >>> Moreover, if I'm not wrong, viewing or parsing such as-is data would >>> actually require replaying the control sequences, to reproduce the >>> actual console contents as it was recorded, which would be quite >>> cumbersome. >> >> It *is* useless, but most of the times the user won't be scrolling at >> all anyways, that's why I don't think these few extra bytes would be a >> problem. > > Quite frankly, when it comes to computing in general, taking the "ah, it > probably won't happen" approach isn't the best thing to do. We should > aim toward covering properly even the edge cases, if you agree. You're absolutely right; however, I did not say "it won't happen", I said that Linux userspace can go deal with it if it happens, and if it wants to. And that in my opinion, the best course of action for U-Boot is to record *everything* (as it does now), and then userspace can decide to filter it for eg. ANSI codes. But I think it's best to keep the U-Boot code dead simple, and also supply the most data to userspace. >> That's what I read as well. Is there support for U-boot to write and >> Linux to read PStores? > > No and yes, but U-Boot can already read pstore. Please see > doc/usage/cmd/pstore.rst for the U-Boot part, and > Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. Irrelevant, as we only want to write out the console log to U-Boot, and not the other way around (that's for collecting panic logs, which is already implemented). > Another benefit of using pstore would be no permanently wasted RAM for > the recorded console contents. Also, having the data recorded to a > storage device also goes along with providing permanent records. I'm positively *not gonna save boot logs to disk*, as most embedded systems have Flash-based storage media; writing to them on every boot would be devastating. Plus, I don't want the console subsystem to depend on any file/disk operations/drivers. Therefore, it will be necessary to permanently keep the logs in-memory, regardless of whether we use the PStore format or something else. Unless Linux wants to copy it elsewhere (which probably will be memory anyways). But maybe we could offer some option in the Linux driver to release the logs, along with the associated memory region... All in all, I'm all for using pre-existing standard interfaces instead of inventing something new and custom, but as I see it now, PStore just cannot do what we want here. If any of you think otherwise, feel free to try to convince me further, or even guide me on how to use PStores. Here's what I would want: * writing already supported in U-Boot * pure in-memory solution * minimal overhead to the console subsystem
Re: Passing boot logs to Linux?
On 2023-12-20 21:03, Sean Anderson wrote: On 12/20/23 14:24, Csókás Bence wrote: On 2023. 12. 20. 9:29, Dragan Simic wrote: On 2023-12-20 08:52, Csókás Bence wrote: On 2023. 12. 20. 8:09, Dragan Simic wrote: On 2023-12-20 07:49, Csókás Bence wrote: That's what I read as well. Is there support for U-boot to write and Linux to read PStores? No and yes, but U-Boot can already read pstore. Please see doc/usage/cmd/pstore.rst for the U-Boot part, and Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. Irrelevant, as we only want to write out the console log to U-Boot, and not the other way around (that's for collecting panic logs, which is already implemented). Another benefit of using pstore would be no permanently wasted RAM for the recorded console contents. Also, having the data recorded to a storage device also goes along with providing permanent records. I'm positively *not gonna save boot logs to disk*, as most embedded systems have Flash-based storage media; writing to them on every boot would be devastating. Plus, I don't want the console subsystem to depend on any file/disk operations/drivers. pstore is agnostic to the backend used. In U-Boot we only support RAM backends at the moment. That's a very good point.
Re: Passing boot logs to Linux?
On 12/20/23 14:24, Csókás Bence wrote: On 2023. 12. 20. 9:29, Dragan Simic wrote: On 2023-12-20 08:52, Csókás Bence wrote: On 2023. 12. 20. 8:09, Dragan Simic wrote: On 2023-12-20 07:49, Csókás Bence wrote: I don't think that would be a huge problem, Linux userspace can filter ANSI control codes if it wants to. For now, I'd like a byte-for-byte copy of the console, as-is, presented to Linux. But wouldn't recording the control sequences as-is actually be pretty much useless? For example, some user can spend a few minutes scrolling through the boot menu, which would produce a fair amount of nearly useless recorded data. Moreover, if I'm not wrong, viewing or parsing such as-is data would actually require replaying the control sequences, to reproduce the actual console contents as it was recorded, which would be quite cumbersome. It *is* useless, but most of the times the user won't be scrolling at all anyways, that's why I don't think these few extra bytes would be a problem. Quite frankly, when it comes to computing in general, taking the "ah, it probably won't happen" approach isn't the best thing to do. We should aim toward covering properly even the edge cases, if you agree. You're absolutely right; however, I did not say "it won't happen", I said that Linux userspace can go deal with it if it happens, and if it wants to. And that in my opinion, the best course of action for U-Boot is to record *everything* (as it does now), and then userspace can decide to filter it for eg. ANSI codes. But I think it's best to keep the U-Boot code dead simple, and also supply the most data to userspace. That's what I read as well. Is there support for U-boot to write and Linux to read PStores? No and yes, but U-Boot can already read pstore. Please see doc/usage/cmd/pstore.rst for the U-Boot part, and Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. Irrelevant, as we only want to write out the console log to U-Boot, and not the other way around (that's for collecting panic logs, which is already implemented). Another benefit of using pstore would be no permanently wasted RAM for the recorded console contents. Also, having the data recorded to a storage device also goes along with providing permanent records. I'm positively *not gonna save boot logs to disk*, as most embedded systems have Flash-based storage media; writing to them on every boot would be devastating. Plus, I don't want the console subsystem to depend on any file/disk operations/drivers. pstore is agnostic to the backend used. In U-Boot we only support RAM backends at the moment. --Sean
Re: Passing boot logs to Linux?
On 2023-12-20 20:24, Csókás Bence wrote: On 2023. 12. 20. 9:29, Dragan Simic wrote: On 2023-12-20 08:52, Csókás Bence wrote: On 2023. 12. 20. 8:09, Dragan Simic wrote: On 2023-12-20 07:49, Csókás Bence wrote: I don't think that would be a huge problem, Linux userspace can filter ANSI control codes if it wants to. For now, I'd like a byte-for-byte copy of the console, as-is, presented to Linux. But wouldn't recording the control sequences as-is actually be pretty much useless? For example, some user can spend a few minutes scrolling through the boot menu, which would produce a fair amount of nearly useless recorded data. Moreover, if I'm not wrong, viewing or parsing such as-is data would actually require replaying the control sequences, to reproduce the actual console contents as it was recorded, which would be quite cumbersome. It *is* useless, but most of the times the user won't be scrolling at all anyways, that's why I don't think these few extra bytes would be a problem. Quite frankly, when it comes to computing in general, taking the "ah, it probably won't happen" approach isn't the best thing to do. We should aim toward covering properly even the edge cases, if you agree. You're absolutely right; however, I did not say "it won't happen", I said that Linux userspace can go deal with it if it happens, and if it wants to. And that in my opinion, the best course of action for U-Boot is to record *everything* (as it does now), and then userspace can decide to filter it for eg. ANSI codes. But I think it's best to keep the U-Boot code dead simple, and also supply the most data to userspace. Hmm, maybe you're right, but I'd have to think a bit more about it. I'd also have to go through the relevant parts of the U-Boot source code in detail. That's what I read as well. Is there support for U-boot to write and Linux to read PStores? No and yes, but U-Boot can already read pstore. Please see doc/usage/cmd/pstore.rst for the U-Boot part, and Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. Irrelevant, as we only want to write out the console log to U-Boot, and not the other way around (that's for collecting panic logs, which is already implemented). Quite frankly, this is a bit confusing to me. Why should it be irrelevant, if we want to store the U-Boot console logs into the pstore, and read them later in Linux? That's exactly what you asked about, and it was a good question. Another benefit of using pstore would be no permanently wasted RAM for the recorded console contents. Also, having the data recorded to a storage device also goes along with providing permanent records. I'm positively *not gonna save boot logs to disk*, as most embedded systems have Flash-based storage media; writing to them on every boot would be devastating. Writing 30-40 KB of data to an even remotely modern flash-based storage media on each boot is pretty much nothing. OTOH, writing 30-40 KB of data to an SPI chip on each boot would be rather bad, but we aren't going to do that for sure. To put things into perspective, I'm writing this on a Pinebook Pro with an eMMC chip, and it has stored many GBs of data over a few years. Even high-endurance microSD cards are rather reliable in that regard. Plus, I don't want the console subsystem to depend on any file/disk operations/drivers. Well, the console would still work as usual even if logging to disk would fail for any reason, which is similar to the serial console still working if the graphical console fails. Moreover, if the disk fails, the system isn't be able to boot, so any RAM-based console logs would be lost in that case. All this makes the RAM-based logging no more resilient to disk failures. Therefore, it will be necessary to permanently keep the logs in-memory, regardless of whether we use the PStore format or something else. Unless Linux wants to copy it elsewhere (which probably will be memory anyways). But maybe we could offer some option in the Linux driver to release the logs, along with the associated memory region... I still think that using disk-based pstore is a better option. Just as you don't want to wear out your flash disks with 30-40 KB of data, I also don't want to waste 30-40 KB of RAM. Also, embedded devices such as the PinePhone would benefit additionally from using disk-based logging, because people often can't do anything in the field if booting fails for some reason, so RAM-based logs would end up being lost, preventing any post-mortem debugging later. Perhaps we can actually offer both options, i.e. both logging to RAM and to disk, so everyone can choose what they prefer. I'd also be fine with working on such a dual approach, but it would obviously be more complex. All in all, I'm all for using pre-existing standard interfaces instead of inventing something new and custom, but as I see it now, PStore just cannot do what we want here. If any
Re: Passing boot logs to Linux?
On 2023-12-20 11:31, Mark Kettenis wrote: Date: Tue, 19 Dec 2023 23:45:48 -0500 From: Sean Anderson On 12/19/23 23:11, Simon Glass wrote: > On Tue, 19 Dec 2023 at 13:15, Csókás Bence wrote: >> Is passing the U-Boot boot log to Linux supported yet? We are working >> with a third-party solution, which works, but is a bit hacky, so I was >> wondering if an official solution has been merged yet. >> >> I saw that there was an option CONFIG_CONSOLE_RECORD that saves >> everything to a membuff, but I don't know if that can be exported to >> Linux yet. And if not in the tree yet, would such a patch be welcome? > > Not yet, but yes I would like to see this. I think most of the infrastructure is already here. We could use either console recording (as mentioned above), which is more complete, or a new LOG_DRIVER (which would have the advantage of omitting things like countdowns). In terms of sending things to Linux, I think the natural choice would be pstore, which we already have a parser for. Well, this should be OS-agnostic of course ;) If possible, of course. What is pstore? Is this documented somewhere? As I already explained briefly in one of my earlier posts, it's a mechanism that the Linux kernel uses to store the kernel crash data on persistent storage, for later examination. Please see the following links, for exaple, for more details: - https://docs.kernel.org/admin-guide/pstore-blk.html - https://man7.org/linux/man-pages/man8/systemd-pstore.8.html - https://blogs.oracle.com/linux/post/pstore-linux-kernel-persistent-storage-file-system
Re: Passing boot logs to Linux?
On 2023. 12. 20. 8:09, Dragan Simic wrote: > On 2023-12-20 07:49, Csókás Bence wrote: >> On 2023. 12. 19. 21:33, Daniel Golle wrote: >>> What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI >>> sequences during menu or count-down before boot, so we'd have to either >>> introduce a dedicated log_printf() call and use that when ever we want >>> the output to also become part of the log buffer or we could somehow >>> filter the recorded console, eliminating all terminal control >>> sequences. >> >> I don't think that would be a huge problem, Linux userspace can filter >> ANSI control codes if it wants to. For now, I'd like a byte-for-byte >> copy of the console, as-is, presented to Linux. > > But wouldn't recording the control sequences as-is actually be pretty > much useless? For example, some user can spend a few minutes > scrolling through the boot menu, which would produce a fair amount of > nearly useless recorded data. > > Moreover, if I'm not wrong, viewing or parsing such as-is data would > actually require replaying the control sequences, to reproduce the > actual console contents as it was recorded, which would be quite > cumbersome. It *is* useless, but most of the times the user won't be scrolling at all anyways, that's why I don't think these few extra bytes would be a problem. And no, you don't need anything to "process" ANSI codes, you can just `cat` them and your terminal will parse it anyways. >> As I undersand (never used it before, just quickly read the kernel doc), >> PStore is for Linux -> U-Boot communication. What I'm doing is the >> opposite. But the approach I envisioned is similar: make U-Boot save a >> copy of the log to a struct at a fixed memory address, then pass this >> address + the length of the log to the kernel via DT and/or commandline. > > Actually, pstore is primarily intended as a means to record and store > kernel crash dumps for later investigation. In other words, the > kernel writes data to pstore during an oops so it can be investigated > later. > > One of the benefits of using pstore for record the U-Boot console > contents, instead of using a memory region, is that no special > utilities would be required to access the recorded console outputs. > Also, many of the requirements that you listed below would be already > fulfilled. > >> What I thought to include in the struct: >> >> * magic cookie (it was in the 3rd party solution, and it's an easy way >> of doing some form of integrity check) >> * struct version (in case we ever change the layout) >> * U-Boot version code (maybe not necessary? It can be parsed out of the >> log as well) >> * length of the log (or it could also be passed via DT/cmdline) >> * and then the log buffer of course > That's what I read as well. Is there support for U-boot to write and Linux to read PStores? Bence
Re: Passing boot logs to Linux?
On 2023. 12. 19. 21:33, Daniel Golle wrote: > Hi Bence, > > On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: >> Hi! >> >> Is passing the U-Boot boot log to Linux supported yet? We are working >> with a third-party solution, which works, but is a bit hacky, so I was >> wondering if an official solution has been merged yet. >> >> I saw that there was an option CONFIG_CONSOLE_RECORD that saves >> everything to a membuff, but I don't know if that can be exported to >> Linux yet. And if not in the tree yet, would such a patch be welcome? > To me this sounds very useful and definitely something we'd like to > have in OpenWrt. So I'd volunteer to review and test your patches once > they are ready. > > What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI > sequences during menu or count-down before boot, so we'd have to either > introduce a dedicated log_printf() call and use that when ever we want > the output to also become part of the log buffer or we could somehow > filter the recorded console, eliminating all terminal control sequences. > > > Cheers > > > Daniel > I don't think that would be a huge problem, Linux userspace can filter ANSI control codes if it wants to. For now, I'd like a byte-for-byte copy of the console, as-is, presented to Linux. On 2023. 12. 20. 6:04, Dragan Simic wrote: > On 2023-12-20 05:45, Sean Anderson wrote: >> On 12/19/23 23:11, Simon Glass wrote: >>> On Tue, 19 Dec 2023 at 13:15, Csókás Bence >>> wrote: Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? >>> >>> Not yet, but yes I would like to see this. >> >> I think most of the infrastructure is already here. We could use either >> console recording (as mentioned above), which is more complete, or a new >> LOG_DRIVER (which would have the advantage of omitting things like >> countdowns). >> >> In terms of sending things to Linux, I think the natural choice would >> be pstore, >> which we already have a parser for. > > Yes, using pstore would make sense. > > As a note, I'm willing to work on this, if Csókás actually doesn't > pick it up. I'll be so happy to see this feature reaching all kinds > of boards and devices. > As I undersand (never used it before, just quickly read the kernel doc), PStore is for Linux -> U-Boot communication. What I'm doing is the opposite. But the approach I envisioned is similar: make U-Boot save a copy of the log to a struct at a fixed memory address, then pass this address + the length of the log to the kernel via DT and/or commandline. What I thought to include in the struct: * magic cookie (it was in the 3rd party solution, and it's an easy way of doing some form of integrity check) * struct version (in case we ever change the layout) * U-Boot version code (maybe not necessary? It can be parsed out of the log as well) * length of the log (or it could also be passed via DT/cmdline) * and then the log buffer of course Bence
Re: Passing boot logs to Linux?
> Date: Tue, 19 Dec 2023 23:45:48 -0500 > From: Sean Anderson > > On 12/19/23 23:11, Simon Glass wrote: > > Hi Csókás, > > > > On Tue, 19 Dec 2023 at 13:15, Csókás Bence wrote: > >> > >> Hi! > >> > >> Is passing the U-Boot boot log to Linux supported yet? We are working > >> with a third-party solution, which works, but is a bit hacky, so I was > >> wondering if an official solution has been merged yet. > >> > >> I saw that there was an option CONFIG_CONSOLE_RECORD that saves > >> everything to a membuff, but I don't know if that can be exported to > >> Linux yet. And if not in the tree yet, would such a patch be welcome? > > > > Not yet, but yes I would like to see this. > > I think most of the infrastructure is already here. We could use > either console recording (as mentioned above), which is more > complete, or a new LOG_DRIVER (which would have the advantage of > omitting things like countdowns). > > In terms of sending things to Linux, I think the natural choice > would be pstore, which we already have a parser for. Well, this should be OS-agnostic of course ;) What is pstore? Is this documented somewhere?
Re: Passing boot logs to Linux?
On 2023-12-20 08:52, Csókás Bence wrote: On 2023. 12. 20. 8:09, Dragan Simic wrote: On 2023-12-20 07:49, Csókás Bence wrote: On 2023. 12. 19. 21:33, Daniel Golle wrote: What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI sequences during menu or count-down before boot, so we'd have to either introduce a dedicated log_printf() call and use that when ever we want the output to also become part of the log buffer or we could somehow filter the recorded console, eliminating all terminal control sequences. I don't think that would be a huge problem, Linux userspace can filter ANSI control codes if it wants to. For now, I'd like a byte-for-byte copy of the console, as-is, presented to Linux. But wouldn't recording the control sequences as-is actually be pretty much useless? For example, some user can spend a few minutes scrolling through the boot menu, which would produce a fair amount of nearly useless recorded data. Moreover, if I'm not wrong, viewing or parsing such as-is data would actually require replaying the control sequences, to reproduce the actual console contents as it was recorded, which would be quite cumbersome. It *is* useless, but most of the times the user won't be scrolling at all anyways, that's why I don't think these few extra bytes would be a problem. Quite frankly, when it comes to computing in general, taking the "ah, it probably won't happen" approach isn't the best thing to do. We should aim toward covering properly even the edge cases, if you agree. And no, you don't need anything to "process" ANSI codes, you can just `cat` them and your terminal will parse it anyways. In that case, you'd probably see only the end results of the control sequences, which makes logging them as-is quite pointless. To see what actually happened on the U-Boot console, you'd need to execute the control sequences step by step and watch the results. As I undersand (never used it before, just quickly read the kernel doc), PStore is for Linux -> U-Boot communication. What I'm doing is the opposite. But the approach I envisioned is similar: make U-Boot save a copy of the log to a struct at a fixed memory address, then pass this address + the length of the log to the kernel via DT and/or commandline. Actually, pstore is primarily intended as a means to record and store kernel crash dumps for later investigation. In other words, the kernel writes data to pstore during an oops so it can be investigated later. One of the benefits of using pstore for record the U-Boot console contents, instead of using a memory region, is that no special utilities would be required to access the recorded console outputs. Also, many of the requirements that you listed below would be already fulfilled. What I thought to include in the struct: * magic cookie (it was in the 3rd party solution, and it's an easy way of doing some form of integrity check) * struct version (in case we ever change the layout) * U-Boot version code (maybe not necessary? It can be parsed out of the log as well) * length of the log (or it could also be passed via DT/cmdline) * and then the log buffer of course That's what I read as well. Is there support for U-boot to write and Linux to read PStores? No and yes, but U-Boot can already read pstore. Please see doc/usage/cmd/pstore.rst for the U-Boot part, and Documentation/admin-guide/pstore-blk.rst for the Linux kernel part. Another benefit of using pstore would be no permanently wasted RAM for the recorded console contents. Also, having the data recorded to a storage device also goes along with providing permanent records.
Re: Passing boot logs to Linux?
On 2023-12-20 07:49, Csókás Bence wrote: On 2023. 12. 19. 21:33, Daniel Golle wrote: On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? To me this sounds very useful and definitely something we'd like to have in OpenWrt. So I'd volunteer to review and test your patches once they are ready. What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI sequences during menu or count-down before boot, so we'd have to either introduce a dedicated log_printf() call and use that when ever we want the output to also become part of the log buffer or we could somehow filter the recorded console, eliminating all terminal control sequences. I don't think that would be a huge problem, Linux userspace can filter ANSI control codes if it wants to. For now, I'd like a byte-for-byte copy of the console, as-is, presented to Linux. But wouldn't recording the control sequences as-is actually be pretty much useless? For example, some user can spend a few minutes scrolling through the boot menu, which would produce a fair amount of nearly useless recorded data. Moreover, if I'm not wrong, viewing or parsing such as-is data would actually require replaying the control sequences, to reproduce the actual console contents as it was recorded, which would be quite cumbersome. On 2023. 12. 20. 6:04, Dragan Simic wrote: On 2023-12-20 05:45, Sean Anderson wrote: On 12/19/23 23:11, Simon Glass wrote: On Tue, 19 Dec 2023 at 13:15, Csókás Bence wrote: Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? Not yet, but yes I would like to see this. I think most of the infrastructure is already here. We could use either console recording (as mentioned above), which is more complete, or a new LOG_DRIVER (which would have the advantage of omitting things like countdowns). In terms of sending things to Linux, I think the natural choice would be pstore, which we already have a parser for. Yes, using pstore would make sense. As a note, I'm willing to work on this, if Csókás actually doesn't pick it up. I'll be so happy to see this feature reaching all kinds of boards and devices. As I undersand (never used it before, just quickly read the kernel doc), PStore is for Linux -> U-Boot communication. What I'm doing is the opposite. But the approach I envisioned is similar: make U-Boot save a copy of the log to a struct at a fixed memory address, then pass this address + the length of the log to the kernel via DT and/or commandline. Actually, pstore is primarily intended as a means to record and store kernel crash dumps for later investigation. In other words, the kernel writes data to pstore during an oops so it can be investigated later. One of the benefits of using pstore for record the U-Boot console contents, instead of using a memory region, is that no special utilities would be required to access the recorded console outputs. Also, many of the requirements that you listed below would be already fulfilled. What I thought to include in the struct: * magic cookie (it was in the 3rd party solution, and it's an easy way of doing some form of integrity check) * struct version (in case we ever change the layout) * U-Boot version code (maybe not necessary? It can be parsed out of the log as well) * length of the log (or it could also be passed via DT/cmdline) * and then the log buffer of course
Re: Passing boot logs to Linux?
On 2023-12-20 05:45, Sean Anderson wrote: On 12/19/23 23:11, Simon Glass wrote: On Tue, 19 Dec 2023 at 13:15, Csókás Bence wrote: Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? Not yet, but yes I would like to see this. I think most of the infrastructure is already here. We could use either console recording (as mentioned above), which is more complete, or a new LOG_DRIVER (which would have the advantage of omitting things like countdowns). In terms of sending things to Linux, I think the natural choice would be pstore, which we already have a parser for. Yes, using pstore would make sense. As a note, I'm willing to work on this, if Csókás actually doesn't pick it up. I'll be so happy to see this feature reaching all kinds of boards and devices.
Re: Passing boot logs to Linux?
On 12/19/23 23:11, Simon Glass wrote: Hi Csókás, On Tue, 19 Dec 2023 at 13:15, Csókás Bence wrote: Hi! Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? Not yet, but yes I would like to see this. I think most of the infrastructure is already here. We could use either console recording (as mentioned above), which is more complete, or a new LOG_DRIVER (which would have the advantage of omitting things like countdowns). In terms of sending things to Linux, I think the natural choice would be pstore, which we already have a parser for. --Sean
Re: Passing boot logs to Linux?
Hi Csókás, On Tue, 19 Dec 2023 at 13:15, Csókás Bence wrote: > > Hi! > > Is passing the U-Boot boot log to Linux supported yet? We are working > with a third-party solution, which works, but is a bit hacky, so I was > wondering if an official solution has been merged yet. > > I saw that there was an option CONFIG_CONSOLE_RECORD that saves > everything to a membuff, but I don't know if that can be exported to > Linux yet. And if not in the tree yet, would such a patch be welcome? Not yet, but yes I would like to see this. Regards, Simon
Re: Passing boot logs to Linux?
+Heinrich Schuchardt
On Tue, 19 Dec 2023 at 13:33, Daniel Golle wrote:
>
> Hi Bence,
>
> On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote:
> > Hi!
> >
> > Is passing the U-Boot boot log to Linux supported yet? We are working
> > with a third-party solution, which works, but is a bit hacky, so I was
> > wondering if an official solution has been merged yet.
> >
> > I saw that there was an option CONFIG_CONSOLE_RECORD that saves
> > everything to a membuff, but I don't know if that can be exported to
> > Linux yet. And if not in the tree yet, would such a patch be welcome?
>
> To me this sounds very useful and definitely something we'd like to
> have in OpenWrt. So I'd volunteer to review and test your patches once
> they are ready.
>
> What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI
> sequences during menu or count-down before boot, so we'd have to either
> introduce a dedicated log_printf() call and use that when ever we want
> the output to also become part of the log buffer or we could somehow
> filter the recorded console, eliminating all terminal control sequences.
I have been asking for this for a while. We need some sort of inline
function to enable ANSI output. Perhaps something like this?
#if CONFIG_IS_ENABLED(ANSI)
bool use_ansi(void);
#else
static inline bool use_ansi(void) { return false; }
#endif
bool use_ansi(void)
{
return gd->flags & GD_FLG_ANSI;
}
>
>
> Cheers
>
>
> Daniel
Regards,
SImon
Re: Passing boot logs to Linux?
On 2023-12-19 21:33, Daniel Golle wrote: On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? To me this sounds very useful and definitely something we'd like to have in OpenWrt. So I'd volunteer to review and test your patches once they are ready. I'd also support having this feature. It would be very, very useful. What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI sequences during menu or count-down before boot, so we'd have to either introduce a dedicated log_printf() call and use that when ever we want the output to also become part of the log buffer or we could somehow filter the recorded console, eliminating all terminal control sequences. Hmm, that's one of the associated challenges. Perhaps those intermediate messages, such as the countdowns and menus, shouldn't be recorded into the memory buffer at all, but just the final outcomes, simply because that's the useful part.
Re: Passing boot logs to Linux?
Hi Bence, On Tue, Dec 19, 2023 at 08:08:48PM +, Csókás Bence wrote: > Hi! > > Is passing the U-Boot boot log to Linux supported yet? We are working > with a third-party solution, which works, but is a bit hacky, so I was > wondering if an official solution has been merged yet. > > I saw that there was an option CONFIG_CONSOLE_RECORD that saves > everything to a membuff, but I don't know if that can be exported to > Linux yet. And if not in the tree yet, would such a patch be welcome? To me this sounds very useful and definitely something we'd like to have in OpenWrt. So I'd volunteer to review and test your patches once they are ready. What comes to mind is that CONFIG_CONSOLE_RECORD also captures ANSI sequences during menu or count-down before boot, so we'd have to either introduce a dedicated log_printf() call and use that when ever we want the output to also become part of the log buffer or we could somehow filter the recorded console, eliminating all terminal control sequences. Cheers Daniel
Passing boot logs to Linux?
Hi! Is passing the U-Boot boot log to Linux supported yet? We are working with a third-party solution, which works, but is a bit hacky, so I was wondering if an official solution has been merged yet. I saw that there was an option CONFIG_CONSOLE_RECORD that saves everything to a membuff, but I don't know if that can be exported to Linux yet. And if not in the tree yet, would such a patch be welcome? Bence
