Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console
On Mon, 2007-10-08 at 10:41 -0400, Steven Rostedt wrote: > -- > On Mon, 8 Oct 2007, Gregory Haskins wrote: > > > > Hi Steve, > > What you describe is exactly what I did. The IRQF_NODELAY handler > > just minimally checks to see if the character is a sysrq related one (or > > KDB, if you have the KDB patches applied). If it is not, it puts the > > character into a ring-buffer and wakes a (kthread-based) tasklet to do > > the normal serial/tty rx path by processing the ring. > > Ah, that was in the second patch (I didn't look at that. My fault). I > didn't see this description in the comments. Well, the lack of a proper description is my fault. I can fix this if there is interest in actually including it. > But you are right, it is very > close to what I did too. But before we add this to -rt, it will need to > probably need to be tested quite a bit more. Totally agree. Note that it already is configurably inert, and defaults to disabled so that might make it work for the general user just fine. FWIW, our "debug" kernels have been running with the feature on for quite a few weeks without problems. But I would fully expect someone external to want to quantify that claim on their own before it would be considered. So give it a whirl if you get a chance. Regards, -Greg - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console
-- On Mon, 8 Oct 2007, Gregory Haskins wrote: > > Hi Steve, > What you describe is exactly what I did. The IRQF_NODELAY handler > just minimally checks to see if the character is a sysrq related one (or > KDB, if you have the KDB patches applied). If it is not, it puts the > character into a ring-buffer and wakes a (kthread-based) tasklet to do > the normal serial/tty rx path by processing the ring. Ah, that was in the second patch (I didn't look at that. My fault). I didn't see this description in the comments. But you are right, it is very close to what I did too. But before we add this to -rt, it will need to probably need to be tested quite a bit more. Especially since this is more to help us developers than to help end users. I would like to see something like this eventually make it into RT. Just because there has been too many times that this would be useful. For now I just use the NMI watchdog and soft lockup detect. But those are very limiting. > > Hope that helps to clarify. Yes, thanks! -- Steve - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console
On Mon, 2007-10-08 at 10:10 -0400, Steven Rostedt wrote: > This issue has hit me enough times where I've played with a few other > ideas. I just haven't had the time to finish them. The main problem is if > the system locks up somewhere we have a lock held that keeps us from > scheduling. Once that happens in -rt, we are as good as dead. Since you > can't get much info out without a JTAG or some other mechanism. > > One thought I had was to create a small handler that could read the serial > and place data into a buffer. This handler would be a IRQ_NODELAY, and it > would wake up a serial thread to handle the data just like it would for > normal serial output. But the serial thread would just read from the > buffer instead of the serial itself. In the case of a sysrq key coming > in, the IRQ_NODELAY handler would handle it. > Hi Steve, What you describe is exactly what I did. The IRQF_NODELAY handler just minimally checks to see if the character is a sysrq related one (or KDB, if you have the KDB patches applied). If it is not, it puts the character into a ring-buffer and wakes a (kthread-based) tasklet to do the normal serial/tty rx path by processing the ring. Hope that helps to clarify. Regards, -Greg - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console
-- On Fri, 5 Oct 2007, Thomas Gleixner wrote: > On Fri, 5 Oct 2007, Gregory Haskins wrote: > > This series may help debugging certain circumstances where the serial > > console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It > > changes > > the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq > > context instead of a kthread. > > > > It works pretty well on this end, though it is admitted not fully baked. > > For > > instance, a few paths in sysrq can still call into non-raw spinlocks > > (sched_debug_show, for instance) which will cause subsequent errors. Also, > > if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I > > may > > send a KDB related patch seperately. > > > > I am sending this out now in case it is helpful to someone. > > The simple non-patch solution is to up the priority of the serial > console irq to maximum. That's usually sufficient to catch runnaway tasks > etc. It does not interfere with the system in normal operation as long as > you do not hit keys in your minicom. > > I doubt that your patch has a chance to survive lockdep and anything which > is not a sysrq. This issue has hit me enough times where I've played with a few other ideas. I just haven't had the time to finish them. The main problem is if the system locks up somewhere we have a lock held that keeps us from scheduling. Once that happens in -rt, we are as good as dead. Since you can't get much info out without a JTAG or some other mechanism. One thought I had was to create a small handler that could read the serial and place data into a buffer. This handler would be a IRQ_NODELAY, and it would wake up a serial thread to handle the data just like it would for normal serial output. But the serial thread would just read from the buffer instead of the serial itself. In the case of a sysrq key coming in, the IRQ_NODELAY handler would handle it. Just a thought. Maybe someday I can get time to play more with this idea, or someone else can come up with one. But to convert the entire handler into nodelay is a bit extreme. -- Steve - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console
On Fri, 2007-10-05 at 18:41 +0200, Thomas Gleixner wrote: > On Fri, 5 Oct 2007, Gregory Haskins wrote: > > This series may help debugging certain circumstances where the serial > > console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It > > changes > > the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq > > context instead of a kthread. > > > > It works pretty well on this end, though it is admitted not fully baked For > > instance, a few paths in sysrq can still call into non-raw spinlocks > > (sched_debug_show, for instance) which will cause subsequent errors. Also, > > if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I > > may > > send a KDB related patch seperately. > > > > I am sending this out now in case it is helpful to someone. > > The simple non-patch solution is to up the priority of the serial > console irq to maximum. That's usually sufficient to catch runnaway tasks > etc. It does not interfere with the system in normal operation as long as > you do not hit keys in your minicom. > > I doubt that your patch has a chance to survive lockdep and anything which > is not a sysrq. Your point is valid, and I thought of this too. The one problem with it is that it is not immune to problems in the scheduler itself (which at the time, I thought I was chasing..turned out to be lockdep), but that is probably a rarity. In any case, we have been using it on a number of debug systems here for a few weeks and the console continues to work well in all functions besides sysrq, believe it or not (*). But I don't see this as being anything used for real. Just thought I would throw it out there in case it was useful or inspired someone to give it some love. Regards, -Greg (*) The trick is that once it figures out its not for sysrq, it schedules a (kthread based) tasklet to do the actual tty() callbacks. signature.asc Description: This is a digitally signed message part
Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console
On Fri, 5 Oct 2007, Gregory Haskins wrote: > This series may help debugging certain circumstances where the serial > console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It changes > the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq > context instead of a kthread. > > It works pretty well on this end, though it is admitted not fully baked. For > instance, a few paths in sysrq can still call into non-raw spinlocks > (sched_debug_show, for instance) which will cause subsequent errors. Also, > if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I may > send a KDB related patch seperately. > > I am sending this out now in case it is helpful to someone. The simple non-patch solution is to up the priority of the serial console irq to maximum. That's usually sufficient to catch runnaway tasks etc. It does not interfere with the system in normal operation as long as you do not hit keys in your minicom. I doubt that your patch has a chance to survive lockdep and anything which is not a sysrq. tglx - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
