Re: [PATCH] powerpc/powernv: Read OPAL error log and export it through sysfs
Benjamin Herrenschmidt writes: > On Wed, 2014-03-05 at 08:56 +0530, Vasant Hegde wrote: >> On 03/05/2014 07:26 AM, Stewart Smith wrote: >> > Vasant Hegde writes: >> >>> +int64_t opal_send_ack_elog(uint64_t log_id); >> >> >> >> Stewart, >> >> >> >> Why are you creating 64bit log ID when actual ID is 32bit ? >> > >> > IIRC it's what OPAL gives us, even though FSP MBOX spec says 32bit. >> > >> >> Stewart, >> >> Better ask OPAL folks to fix in firmware ? > > Why bother ? That way if future versions of OPAL want to use 64-bit IDs, > they can. Yeah, that was my thought. The number of error logs on some POWER8 systems almost needs 64bit already :) ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH] powerpc/powernv: Read OPAL error log and export it through sysfs
On Wed, 2014-03-05 at 08:56 +0530, Vasant Hegde wrote: > On 03/05/2014 07:26 AM, Stewart Smith wrote: > > Vasant Hegde writes: > >>> +int64_t opal_send_ack_elog(uint64_t log_id); > >> > >> Stewart, > >> > >> Why are you creating 64bit log ID when actual ID is 32bit ? > > > > IIRC it's what OPAL gives us, even though FSP MBOX spec says 32bit. > > > > Stewart, > > Better ask OPAL folks to fix in firmware ? Why bother ? That way if future versions of OPAL want to use 64-bit IDs, they can. Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH] powerpc/powernv: Read OPAL error log and export it through sysfs
On 03/05/2014 07:26 AM, Stewart Smith wrote: Vasant Hegde writes: +int64_t opal_send_ack_elog(uint64_t log_id); Stewart, Why are you creating 64bit log ID when actual ID is 32bit ? IIRC it's what OPAL gives us, even though FSP MBOX spec says 32bit. Stewart, Better ask OPAL folks to fix in firmware ? -Vasant ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH] powerpc/powernv: Read OPAL error log and export it through sysfs
On 03/05/2014 07:26 AM, Stewart Smith wrote: Vasant Hegde writes: +int64_t opal_send_ack_elog(uint64_t log_id); Stewart, Why are you creating 64bit log ID when actual ID is 32bit ? IIRC it's what OPAL gives us, even though FSP MBOX spec says 32bit. Stewart, Better ask OPAL folks to fix in firmware ? -Vasant ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH] powerpc/powernv: Read OPAL error log and export it through sysfs
Vasant Hegde writes: >> +int64_t opal_send_ack_elog(uint64_t log_id); > > Stewart, > > Why are you creating 64bit log ID when actual ID is 32bit ? IIRC it's what OPAL gives us, even though FSP MBOX spec says 32bit. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH] powerpc/powernv: Read OPAL error log and export it through sysfs
On 02/28/2014 06:28 AM, Stewart Smith wrote: Based on a patch by: Mahesh Salgaonkar This patch adds support to read error logs from OPAL and export them to userspace through a sysfs interface. We export each log entry as a directory in /sys/firmware/opal/elog/ Currently, OPAL will buffer up to 128 error log records, we don't need to have any knowledge of this limit on the Linux side as that is actually largely transparent to us. Each error log entry has the following files: id, type, acknowledge, raw. Currently we just export the raw binary error log in the 'raw' attribute. In a future patch, we may parse more of the error log to make it a bit easier for userspace (e.g. to be able to display a brief summary in petitboot without having to have a full parser). If we have >128 logs from OPAL, we'll only be notified of 128 until userspace starts acknowledging them. This limitation may be lifted in the future and with this patch, that should "just work" from the linux side. A userspace daemon should: - wait for error log entries using normal mechanisms (we announce creation) - read error log entry - save error log entry safely to disk - acknowledge the error log entry - rinse, repeat. On the Linux side, we read the error log when we're notified of it. This possibly isn't ideal as it would be better to only read them on-demand. However, this doesn't really work with current OPAL interface, so we read the error log immediately when notified at the moment. I've tested this pretty extensively and am rather confident that the linux side of things works rather well. There is currently an issue with the service processor side of things for >128 error logs though. Signed-off-by: Stewart Smith --- Documentation/ABI/stable/sysfs-firmware-opal-elog | 60 arch/powerpc/include/asm/opal.h | 13 + arch/powerpc/platforms/powernv/Makefile |2 +- arch/powerpc/platforms/powernv/opal-elog.c| 312 + arch/powerpc/platforms/powernv/opal-wrappers.S|5 + arch/powerpc/platforms/powernv/opal.c |2 + 6 files changed, 393 insertions(+), 1 deletion(-) create mode 100644 Documentation/ABI/stable/sysfs-firmware-opal-elog create mode 100644 arch/powerpc/platforms/powernv/opal-elog.c diff --git a/Documentation/ABI/stable/sysfs-firmware-opal-elog b/Documentation/ABI/stable/sysfs-firmware-opal-elog new file mode 100644 index 000..e1f3058 --- /dev/null +++ b/Documentation/ABI/stable/sysfs-firmware-opal-elog @@ -0,0 +1,60 @@ +What: /sys/firmware/opal/elog +Date: Feb 2014 +Contact: Stewart Smith +Description: + This directory exposes error log entries retrieved + through the OPAL firmware interface. + + Each error log is identified by a unique ID and will + exist until explicitly acknowledged to firmware. + + Each log entry has a directory in /sys/firmware/opal/elog. + + Log entries may be purged by the service processor + before retrieved by firmware or retrieved/acknowledged by + Linux if there is no room for more log entries. + + In the event that Linux has retrieved the log entries + but not explicitly acknowledged them to firmware and + the service processor needs more room for log entries, + the only remaining copy of a log message may be in + Linux. + + Typically, a user space daemon will monitor for new + entries, read them out and acknowledge them. + + The service processor may be able to store more log + entries than firmware can, so after you acknowledge + an event from Linux you may instantly get another one + from the queue that was generated some time in the past. + + The raw log format is a binary format. We currently + do not parse this at all in kernel, leaving it up to + user space to solve the problem. In future, we may + do more parsing in kernel and add more files to make + it easier for simple user space processes to extract + more information. + + For each log entry (directory), there are the following + files: + + id: An ASCII representation of the ID of the + error log, in hex - e.g. "0x01". + + type: An ASCII representation of the type id and + description of the type of error log. + Currently just "0x00 PEL" - platform error log. + In the future there may be additional types. + + raw:A read-only binary file that can be read + to get the raw log entry. These are +
Re: [PATCH] powerpc/powernv: Read opal error log and export it through sysfs interface.
Mahesh Jagannath Salgaonkar writes: >> I think we could provide a better interface with instead having a file >> per log message appear in sysfs. We're never going to have more than 128 >> of these at any one time on the Linux side, so it's not going to bee too >> many files. > > It is not just about 128 files, we may be adding/removing sysfs node for > every new log id that gets informed to kernel and ack-ed. In worst case, > when we have flood of elog errors with user daemon consuming it and > ack-ing back to get ready for next log in a tight poll, we may > continuously add/remove the sysfs node for each new . Do we ever get a storm of hundreds/thousands of them though? If many come it at once userspace may just be woken up one or two times, as it would just select() and wait for events. >> I've seen some conflicting things on this - is it 2kb or 16kb? > > We choose 16kb because we want to pull all the log data and not > partial. So the max log size for any one entry is in fact 16kb? >> This means we constantly use 128 * sizeof(struct opal_err_log) which >> equates to somewhere north of 2MB of memory (due to list overhead). >> >> I don't think we need to statically allocate this, we can probably just >> allocate on-demand as in a typical system you're probably quite >> unlikely to have too many of these sitting around (besides, if for >> whatever reason we cannot allocate memory at some point, that's okay >> because we can read it again later). > > The reason we choose to go for static allocation is, we can not afford > to drop or delay a critical error log due to memory allocation failure. > OR we can keep static allocations for critical errors and follow dynamic > allocation for informative error logs. What do you say? Userspace is probably going to have to do IO to get the log and ack it, so it's probably not a huge problem - if we can't allocate a few kb in a couple of attempts then we likely have bigger problems. If we were going to have a sustained amount of hundreds/thousands of these per second then perhaps we'd have other issues, but from what I understand we're probably only going to have a handful per year on a typical system? (I am, of course, not talking about our dev systems, which are rather atypical :) I'll likely have a patch today that shows kind of what I mean. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH] powerpc/powernv: Read opal error log and export it through sysfs interface.
On 02/21/2014 05:41 AM, Stewart Smith wrote:
> Mahesh J Salgaonkar writes:
> > This patch adds support to read error logs from OPAL and export them
> > to userspace through sysfs interface /sys/firmware/opa/opal-elog.
Hi Stewart,
Thanks for the review. This code definitely needs improvement.
>
> I think we could provide a better interface with instead having a file
> per log message appear in sysfs. We're never going to have more than 128
> of these at any one time on the Linux side, so it's not going to bee too
> many files.
It is not just about 128 files, we may be adding/removing sysfs node for
every new log id that gets informed to kernel and ack-ed. In worst case,
when we have flood of elog errors with user daemon consuming it and
ack-ing back to get ready for next log in a tight poll, we may
continuously add/remove the sysfs node for each new .
>
> e.g. /sys/firmware/opal/elog/
>
> that way, any new file in /sys/firmware/opal/elog/ means there's a new
> log entry available. I believe there's
>
> To ack a log, you could just echo 'ack' to the file.
>
> The other option woudl be to more closely follow what sysfs is meant to
> be - ascii text. This would mean having more (any) of the parser in
> kernel for the error logs - which may/may not be a bad idea.
>
> However, it would make the end user code for consuming them much much
> simpler, and that may be a good thing.
>
> Having some way of getting some information out without a userspace
> parser is probably good though, I'm pretty sure having only a binary
> interface in /sys is at least partially frowned upon.
>
> > This is what user space tool would do:
> > - Read error log from /sys/firmware/opa/opal-elog.
> > - Save it to the disk.
> > - Send an acknowledgement on successful consumption by writing error log
> > id to /sys/firmware/opa/opal-elog-ack.
>
> A userspace tool may want to explicitly *not* ack the log too, or only
> ack some entries, so the interface sohuld be sane for this use case too.
>
> e.g. we could display them in petitboot.
>
> > diff --git a/arch/powerpc/platforms/powernv/opal-elog.c
> > b/arch/powerpc/platforms/powernv/opal-elog.c
> [ 2 more citation lines. Click/Enter to show. ]
> > new file mode 100644
> > index 000..fc891ae
> > --- /dev/null
> > +++ b/arch/powerpc/platforms/powernv/opal-elog.c
> > @@ -0,0 +1,309 @@
>
> > +/* Maximum size of a single log on FSP is 16KB */
> > +#define OPAL_MAX_ERRLOG_SIZE 16384
>
> I've seen some conflicting things on this - is it 2kb or 16kb?
We choose 16kb because we want to pull all the log data and not partial.
>
> > +
> > +struct opal_err_log {
> > + struct list_head link;
> > + uint64_t opal_log_id;
>
> why is this uint64_t and not uint32_t? It appears that the log id is 32bits.
Agree, This needs to be changed to uint_32.
>
> > + size_t opal_log_size;
> > + uint8_t data[OPAL_MAX_ERRLOG_SIZE];
> > +};
> > +
> > +/* Pre-allocated temp buffer to pull error log from opal. */
> > +static uint8_t err_log_data[OPAL_MAX_ERRLOG_SIZE];
>
> Why do we need temporary space? Why not just store directly into struct
> opal_err_log?
>
> > +/* Protect err_log_data buf */
> > +static DEFINE_MUTEX(err_log_data_mutex);
> [ 15 more citation lines. Click/Enter to show. ]
> > +
> > +static uint64_t total_log_size;
> > +static bool opal_log_available;
> > +static LIST_HEAD(elog_list);
> > +static LIST_HEAD(elog_ack_list);
> > +
> > +/* lock to protect elog_list and elog-ack_list. */
> > +static DEFINE_SPINLOCK(opal_elog_lock);
> > +
> > +static DECLARE_WAIT_QUEUE_HEAD(opal_log_wait);
> > +
> > +/*
> > + * Interface for user to acknowledge the error log.
> > + *
> > + * Once user acknowledge the log, we delete that record entry from the
> > + * list and move it ack list.
> > + */
> > +void opal_elog_ack(uint64_t ack_id)
>
> s/ack_id/log_id/
Yup. makes sense.
>
> > +
> > +static ssize_t elog_ack_store(struct kobject *kobj,
> [ 7 more citation lines. Click/Enter to show. ]
> > + struct kobj_attribute *attr,
> > + const char *buf, size_t count)
> > +{
> > + uint32_t log_ack_id;
> > + log_ack_id = *(uint32_t *) buf;
> > +
> > + /* send acknowledgment to FSP */
> > + opal_elog_ack(log_ack_id);
> > + return 0;
> > +}
>
> This function has a few problems:
>
> Consider the following actions:
> $ echo 1 > /sys/firmware/opal/opal-elog-ack
> $ echo 'abcde' > /sys/firmware/opal/opal-elog-ack
>
> The former will read undefined memory and the latter will make a kernel
> thread, rsyslogd and systemd-journal all each a CPU each.
>
> Basically, the problems are:
> 1) not endian safe
> 2) not following store API of returning nr bytes read
> 3) binary interface. Use sscanf to read numbers instead.
>
> > +/*
> > + * Show error log records to user.
> [ 9 more citation lines. Click/Enter to show. ]
> > + */
> > +
Re: [PATCH] powerpc/powernv: Read opal error log and export it through sysfs interface.
Mahesh J Salgaonkar writes:
> This patch adds support to read error logs from OPAL and export them
> to userspace through sysfs interface /sys/firmware/opa/opal-elog.
I think we could provide a better interface with instead having a file
per log message appear in sysfs. We're never going to have more than 128
of these at any one time on the Linux side, so it's not going to bee too
many files.
e.g. /sys/firmware/opal/elog/
that way, any new file in /sys/firmware/opal/elog/ means there's a new
log entry available. I believe there's
To ack a log, you could just echo 'ack' to the file.
The other option woudl be to more closely follow what sysfs is meant to
be - ascii text. This would mean having more (any) of the parser in
kernel for the error logs - which may/may not be a bad idea.
However, it would make the end user code for consuming them much much
simpler, and that may be a good thing.
Having some way of getting some information out without a userspace
parser is probably good though, I'm pretty sure having only a binary
interface in /sys is at least partially frowned upon.
> This is what user space tool would do:
> - Read error log from /sys/firmware/opa/opal-elog.
> - Save it to the disk.
> - Send an acknowledgement on successful consumption by writing error log
> id to /sys/firmware/opa/opal-elog-ack.
A userspace tool may want to explicitly *not* ack the log too, or only
ack some entries, so the interface sohuld be sane for this use case too.
e.g. we could display them in petitboot.
> diff --git a/arch/powerpc/platforms/powernv/opal-elog.c
> b/arch/powerpc/platforms/powernv/opal-elog.c
[ 2 more citation lines. Click/Enter to show. ]
> new file mode 100644
> index 000..fc891ae
> --- /dev/null
> +++ b/arch/powerpc/platforms/powernv/opal-elog.c
> @@ -0,0 +1,309 @@
> +/* Maximum size of a single log on FSP is 16KB */
> +#define OPAL_MAX_ERRLOG_SIZE16384
I've seen some conflicting things on this - is it 2kb or 16kb?
> +
> +struct opal_err_log {
> +struct list_head link;
> +uint64_t opal_log_id;
why is this uint64_t and not uint32_t? It appears that the log id is 32bits.
> +size_t opal_log_size;
> +uint8_t data[OPAL_MAX_ERRLOG_SIZE];
> +};
> +
> +/* Pre-allocated temp buffer to pull error log from opal. */
> +static uint8_t err_log_data[OPAL_MAX_ERRLOG_SIZE];
Why do we need temporary space? Why not just store directly into struct
opal_err_log?
> +/* Protect err_log_data buf */
> +static DEFINE_MUTEX(err_log_data_mutex);
[ 15 more citation lines. Click/Enter to show. ]
> +
> +static uint64_t total_log_size;
> +static bool opal_log_available;
> +static LIST_HEAD(elog_list);
> +static LIST_HEAD(elog_ack_list);
> +
> +/* lock to protect elog_list and elog-ack_list. */
> +static DEFINE_SPINLOCK(opal_elog_lock);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(opal_log_wait);
> +
> +/*
> + * Interface for user to acknowledge the error log.
> + *
> + * Once user acknowledge the log, we delete that record entry from the
> + * list and move it ack list.
> + */
> +void opal_elog_ack(uint64_t ack_id)
s/ack_id/log_id/
> +
> +static ssize_t elog_ack_store(struct kobject *kobj,
[ 7 more citation lines. Click/Enter to show. ]
> +struct kobj_attribute *attr,
> +const char *buf, size_t count)
> +{
> +uint32_t log_ack_id;
> +log_ack_id = *(uint32_t *) buf;
> +
> +/* send acknowledgment to FSP */
> +opal_elog_ack(log_ack_id);
> +return 0;
> +}
This function has a few problems:
Consider the following actions:
$ echo 1 > /sys/firmware/opal/opal-elog-ack
$ echo 'abcde' > /sys/firmware/opal/opal-elog-ack
The former will read undefined memory and the latter will make a kernel
thread, rsyslogd and systemd-journal all each a CPU each.
Basically, the problems are:
1) not endian safe
2) not following store API of returning nr bytes read
3) binary interface. Use sscanf to read numbers instead.
> +/*
> + * Show error log records to user.
[ 9 more citation lines. Click/Enter to show. ]
> + */
> +static ssize_t opal_elog_show(struct file *filp, struct kobject *kobj,
> +struct bin_attribute *bin_attr, char *buf,
> +loff_t pos, size_t count)
> +{
> +unsigned long flags;
> +struct opal_err_log *record, *next;
> +size_t size = 0;
> +size_t data_to_copy = 0;
> +int error = 0;
> +
> +/* Display one log@a time. */
use words, not @.
> +if (count > OPAL_MAX_ERRLOG_SIZE)
> +count = OPAL_MAX_ERRLOG_SIZE;
[ 23 more citation lines. Click/Enter to show. ]
> +spin_lock_irqsave(&opal_elog_lock, flags);
> +/* Align the pos to point within total errlog size. */
> +if (total_log_size && pos > total_log_size)
> +pos = pos % total_log_size;
> +
> +/*
> + * if pos goes beyond tot
