On Mon, 2010-08-09 at 22:45 +0400, Vladislav Bolkhovitin wrote:
> David Dillow, on 08/09/2010 06:49 PM wrote:
> > I'm not sure it makes sense to enable/disable this at runtime -- we
> > don't do it for NAPI, why do it for block devices? I'm not even sure I'd
> > want to see a config option for it i
David Dillow, on 08/09/2010 06:49 PM wrote:
On Sun, 2010-08-08 at 20:19 +0200, Bart Van Assche wrote:
On Sat, Aug 7, 2010 at 6:32 PM, Roland Dreier wrote:
Not sure that I follow the problem you're worried about. A given
tasklet can only be running on one CPU at any one time -- if an
interrupt
On Sun, 2010-08-08 at 20:19 +0200, Bart Van Assche wrote:
> On Sat, Aug 7, 2010 at 6:32 PM, Roland Dreier wrote:
> > Not sure that I follow the problem you're worried about. A given
> > tasklet can only be running on one CPU at any one time -- if an
> > interrupt occurs and reschedules the taskle
On Mon, Aug 9, 2010 at 1:51 AM, Jason Gunthorpe
wrote:
>
> [ ... ]
>
> Further, the approach you outline in your follow on message for
> blkio, has problems.. Look at how IPOIB does NAPI to see how
> this must look.
>
> 1) ib_req_notify_cq must only be called if you are processing less
> the bud
On Sun, Aug 08, 2010 at 08:16:55PM +0200, Bart Van Assche wrote:
> On Sun, Aug 8, 2010 at 3:38 AM, Jason Gunthorpe
> wrote:
> > [ ... ]
> >
> > No, all hardware pretty much works like this. The general flow is:
> >
> > IRQ happens
> > ?(if level triggered 'ack' the IRQ to the HW, to suppress the l
On Sat, Aug 7, 2010 at 6:32 PM, Roland Dreier wrote:
> Not sure that I follow the problem you're worried about. A given
> tasklet can only be running on one CPU at any one time -- if an
> interrupt occurs and reschedules the tasklet then it just runs again
> when it exits.
>
> Also I'm not sure I
On Sun, Aug 8, 2010 at 3:38 AM, Jason Gunthorpe
wrote:
> [ ... ]
>
> No, all hardware pretty much works like this. The general flow is:
>
> IRQ happens
> (if level triggered 'ack' the IRQ to the HW, to suppress the level)
> SW processes
> SW 'does something' to the HW to cause new IRQs to happen
On Sat, Aug 07, 2010 at 09:56:13AM +0200, Bart Van Assche wrote:
> The above implies that one must be careful when applying a common
> Linux practice, that is to defer interrupt handling from IRQ context
> to tasklet context. Since tasklets are executed with interrupts
> enabled, invoking ib_req_n
> The above implies that one must be careful when applying a common
> Linux practice, that is to defer interrupt handling from IRQ context
> to tasklet context. Since tasklets are executed with interrupts
> enabled, invoking ib_req_notify_cq(cq, IB_CQ_NEXT_COMP) from tasklet
> context may caus
On Tue, Jul 27, 2010 at 8:20 PM, Jason Gunthorpe
wrote:
> On Tue, Jul 27, 2010 at 08:03:25PM +0200, Bart Van Assche wrote:
>
>> As far as I know it is not possible for a HCA to tell whether or not a
>> CPU has finished executing the interrupt it triggered. So it is not
>> possible for the HCA to i
Roland Dreier wrote:
- If we did get an error, there's not much we can do except keep
polling and try to request notification again later -- exactly the
same thing we would do if we got a positive return value.
Basically, you're right. My only concern is a case where a hw driver
keeps ret
On Wed, 2010-07-28 at 11:16 -0700, Roland Dreier wrote:
> > > Actually, I tried to implement the completion callback
> > > in a workqueue thread but ipoib_cm_handle_tx_wc() calls
> > > netif_tx_lock() which isn't safe unless it is called
> > > from an IRQ handler or netif_tx_lock_bh() is cal
> > Actually, I tried to implement the completion callback
> > in a workqueue thread but ipoib_cm_handle_tx_wc() calls
> > netif_tx_lock() which isn't safe unless it is called
> > from an IRQ handler or netif_tx_lock_bh() is called first.
> Oh, sounds like a bug in IPoIB. I guess we coul
On Wed, 2010-07-28 at 11:05 -0700, Roland Dreier wrote:
> > Actually, I tried to implement the completion callback
> > in a workqueue thread but ipoib_cm_handle_tx_wc() calls
> > netif_tx_lock() which isn't safe unless it is called
> > from an IRQ handler or netif_tx_lock_bh() is called first.
>
> Actually, I tried to implement the completion callback
> in a workqueue thread but ipoib_cm_handle_tx_wc() calls
> netif_tx_lock() which isn't safe unless it is called
> from an IRQ handler or netif_tx_lock_bh() is called first.
Oh, sounds like a bug in IPoIB. I guess we could fix it by jus
On Wed, Jul 28, 2010 at 7:51 PM, Ralph Campbell
wrote:
> On Wed, 2010-07-28 at 10:42 -0700, Roland Dreier wrote:
>> > - Some time ago I observed that the kernel reported soft lockups
>> > because of spin_lock() calls inside a completion handler. These
>> > spinlocks were not locked in any other
Actually, I tried to implement the completion callback
in a workqueue thread but ipoib_cm_handle_tx_wc() calls
netif_tx_lock() which isn't safe unless it is called
from an IRQ handler or netif_tx_lock_bh() is called first.
On Wed, 2010-07-28 at 10:42 -0700, Roland Dreier wrote:
> > - Some time ago
> Roland, I'm wasn't sure if/howmuch the results are buggy, but the
> IPoIB poll loop doesn't check whether the return code of
> ib_req_notify_cq is negative (error) or positive (more completions to
> poll), any thoughts on the matter?
I think I had two things in mind here:
- I don't know of
> - Some time ago I observed that the kernel reported soft lockups
> because of spin_lock() calls inside a completion handler. These
> spinlocks were not locked in any other context than the completion
> handler itself. And the lockups disappeared after having replaced the
> spin_lock() calls
On Tue, Jul 27, 2010 at 09:28:54PM +0200, Bart Van Assche wrote:
> I have two more questions: - Some time ago I observed that the
> kernel reported soft lockups because of spin_lock() calls inside a
> completion handler. These spinlocks were not locked in any other
> context than the completion ha
On Tue, Jul 27, 2010 at 8:20 PM, Jason Gunthorpe
wrote:
>
> On Tue, Jul 27, 2010 at 08:03:25PM +0200, Bart Van Assche wrote:
>
> > As far as I know it is not possible for a HCA to tell whether or not a
> > CPU has finished executing the interrupt it triggered. So it is not
> > possible for the HCA
On Tue, Jul 27, 2010 at 08:03:25PM +0200, Bart Van Assche wrote:
> As far as I know it is not possible for a HCA to tell whether or not a
> CPU has finished executing the interrupt it triggered. So it is not
> possible for the HCA to implement the above requirement by delaying
> the generation of
On Tue, Jul 27, 2010 at 6:50 PM, Roland Dreier wrote:
> [ ... ]
>
> From Documentation/infiniband/core_locking.txt:
>
> The low-level driver is responsible for ensuring that multiple
> completion event handlers for the same CQ are not called
> simultaneously. The driver must guarantee that onl
> In the applications I'm familiar with InfiniBand is being used not
> only because of its low latency but also because of its high
> throughput.
Yes, I seem to recall hearing that people care about throughput as well.
> In order to handle such loads efficiently, interrupts have to be
> spre
On Mon, Jul 26, 2010 at 9:22 PM, Roland Dreier wrote:
> [ ... ]
>
> Another approach is to just always run the completion processing for a
> given CQ on a single CPU and avoid locking entirely. If you want more
> CPUs to spread the work, just use multiple CQs and multiple event vectors.
In the a
Roland Dreier wrote:
> > do {
> > while (ib_poll_cq(cq, 1, &wc) > 0)
> > /* process wc */
> > } while (ib_req_notify_cq(cq, IB_CQ_NEXT_COMP |
> > IB_CQ_REPORT_MISSED_EVENTS) > 0);
> This approach can be used to have race-free in-order processing of
> completions using a scheme suc
> 2. Double completion processing loop
>
> * Initialization:
> ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
>
> * Notification handler:
>
> struct ib_wc wc;
> do {
> while (ib_poll_cq(cq, 1, &wc) > 0)
> /* process wc */
> } while (ib_req_notify_cq(cq, IB_CQ_NEXT_COMP |
> IB_CQ
On Mon, Jul 26, 2010 at 4:21 PM, Steve Wise wrote:
> On 07/25/2010 01:54 PM, Bart Van Assche wrote:
>>
>> [ ... ]
>>
>> The only way I know of to prevent out-of-order completion processing
>> with the current OFED verbs API is to protect the whole completion
>> processing loop against concurrent e
On 07/25/2010 01:54 PM, Bart Van Assche wrote:
One of the most common operations when using the verbs API is to
dequeue and process completions. For many applications, e.g. storage
protocols, processing completions in order is a correctness
requirement. Unfortunately with the current IB verbs API
One of the most common operations when using the verbs API is to
dequeue and process completions. For many applications, e.g. storage
protocols, processing completions in order is a correctness
requirement. Unfortunately with the current IB verbs API it is not
possible to process completions in ord
30 matches
Mail list logo
