Seems to me like interrupt-priorities could help you a lot here. :-)
Yes, doing anything other than the few items mentioned (including
cv_signal!) would be extraordinarily bad in high level interrupt context.
The only reason to use high-level interrupt context is if you have
specific requirements based on a tiny FIFO or hard real-time service
requirements. Few devices have those requirements. (Ancient serial
ports - those with a single 8 bit FIFO -- used to have them, but these
days even serial ports should be able to run at regular priority without
risking FIFO under or overruns.)
-- Garrett
Joachim Worringen wrote:
> Garrett D'Amore schrieb:
>
>> You need to synchronize with a mutex that is initialized with an
>> appropriately high level interrupt block cookie.
>>
>> Anytime is_running is checked, or is changed, that mutex should be
>> held. The mutex basically is implemented in a way that prevents the
>> high level interrupt from running while you hold it. Obviously, you
>> don't want to hold any longer than you absolutely have to, and you
>> shouldn't do anything that requires sleeping, etc. while holding it.
>> (Such locks should always, I believe, be leaf locks...)
>>
>
> Sure, this is all well described in the manuals like
> http://docs.sun.com/app/docs/doc/816-4854/interrupt-18?a=view, and I am
> confident I understand it and am actually using this successfully. But
> the race I'm asking about is present (as I suspect) in exactly this
> example code.
>
>
>> A fundamental question arises though: why do you need hi-level
>> interrupts? You can have "hard interrupts" that are not high level --
>> by default most PCI devices do *not* use high level interrupt handling.
>> If you don't use high level interrupts, then you don't need any of this
>> ugly hand off code to a soft interrupt.
>>
>
> Our PCIe device (Dolphin DXH510, a PCI-Express Interconnect HBA) gets an
> MSI interrupt priority 12 assigned, and it can not be lowered. Don't ask
> me why. Just searched the web a bit (again) and found your related
> posting
> http://mail.opensolaris.org/pipermail/opensolaris-arc/2008-May/008874.html
> where you describe the "interrupt-priorities" property in drivers.conf -
> will try this out!
>
> [BTW: the interrupt-priorities property is at least documented in
> http://developers.sun.com/solaris/developer/support/driver/wps/pci/html/Interrupts.doc.html,
>
> as I just found out.]
>
>
>> As far as synchronization issues go with work queue management, there
>> are several solutions:
>>
>> * kmem_alloc()/zalloc() can be used with KM_NOSLEEP -- in which case
>> you need to be prepared to deal with errors.
>>
>
> I read in a Sun documentation (not "Writing Device Driver", have to look
> it up in the office) that only mutex_enter/_exit (on hi-level mutex),
> ddi_read/_write and soft_intr_trigger were allowed to be called in
> high-level interrupt context. And we got a kernel panic when trying to
> use a kmem_cache allocator (with KM_NOSLEEP).
>
>
>> * you can preallocate what you need, so you can't run out
>>
>
> I can probably allocate sufficient pieces of "work units", all only a
> few bytes, but I can not know for sure in advance how many (thousand)
> interrupts will come in, and how fast the rate of processing them in the
> softintr handler (or taskq) is. I know that allocating "enough" and
> post-allocating as we do usually works (it does for us); but I would
> like to know if there are optimal ways to achieve this.
>
>
>> * if your work queue is relatively fixed, and you don't need to pass
>> data, you can use taskq's to have a different thread. You can avoid
>> having to worry about ddi_taskq_dispatch() by using a master loop and a
>> cv synchronization. (You can use bitfields to indicate which tasks to
>> perform, in such a case.) The sdcard code does this: see
>> "sda_slot_thread()" in usr/src/uts/common/io/sdcard/impl/sda_slot.c for
>> example code.
>>
>
> I'll have a look for inspiration, thanks, but as we are porting
> (generalizing, to be exact) code from Linux with a given
> "schedule_workqueue" functionality, I don't want to change it more than
> really necessary (time matters).
>
> Thanks for your input! Joachim
>
>
>> -- Garrett
>>
>> Joachim Worringen wrote:
>>
>>> Greetings,
>>>
>>> a more generic follow-up question on dispatching soft-interrupts from
>>> hi-level (hard) interrupts: as described in the Sun documentation at
>>> several places (device driver tutorial, or
>>> http://developers.sun.com/solaris/articles/interrupt_handlers.html),
>>> the softintr and hardintr handler synchronize via a "is_running" flag,
>>> protected by a mutex.
>>>
>>> If 'is_running' is true, the softintr handler will not be triggered
>>> (again). However, what about the race between the softintr handler
>>> setting 'is_running' to false (and releasing the lock), and the time
>>> that it is actually returning from the handler function and is
>>> declared "non-running", so that it can be triggered again?
>>>
>>> The hardintr handler may come in in this very moment, assume the
>>> softintr handler is not running, trigger it, get EPENDING and quits =>
>>> an interrupt would be lost.
>>>
>>> Admitted, the race is small - or am I wrong?
>>>
>>> Additionally, what is best practise for the work queue maintained
>>> between these two handlers? As we can not allocate memory within the
>>> hardirq-handler, we need to use some statically allocated queue.
>>> Assuming the worst, this queue could be full when the hardintr handler
>>> wants to queue something => interrupt lost.
>>>
>>> We are trying to prevent this by letting the softintr handler increase
>>> the number of available queue entries depending on a low-water mark,
>>> but still not bullet proof, or is it?
>>>
>>> Joachim
>>>
>>>
>>>
>>>
>
>
>
_______________________________________________
opensolaris-code mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/opensolaris-code
