On Tue, Nov 24, 2015 at 06:47:14PM -0800, Caitlin Bestler wrote:
> Acknowledge packet for the last packet of the request has been
> committed to the wire (including the appropriate fields for RDMA
> READ response).
> The target side cannot generate a response before it
On Tue, Nov 24, 2015 at 10:08:39AM -0700, c...@asomi.com wrote:
>>> My recollection of the IB verbs is that they were unlikely to have
>>> overlooked something like that. If it did slip through then there
>>> should be an errata.
>>verbs reflects the wire protocol and the wire
On 11/24/2015 2:03 AM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote:
Are there actual HCAs that make this mistake?
All IB HCAs have this behavior and require apps to see a send CQ
completion before making any statements about the state of the send Q
On Tue, Nov 24, 2015 at 10:08:39AM -0700, c...@asomi.com wrote:
>>> My recollection of the IB verbs is that they were unlikely to have
>>> overlooked something like that. If it did slip through then there
>>> should be an errata.
>>verbs reflects the wire protocol and the wire
On Tue, Nov 24, 2015 at 06:47:14PM -0800, Caitlin Bestler wrote:
> Acknowledge packet for the last packet of the request has been
> committed to the wire (including the appropriate fields for RDMA
> READ response).
> The target side cannot generate a response before it
On 11/24/2015 2:03 AM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote:
Are there actual HCAs that make this mistake?
All IB HCAs have this behavior and require apps to see a send CQ
completion before making any statements about the state of the send Q
On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote:
> Is it possible for an IB HCA to transmit a response on a QP and not
> in that packet or a previous packet acknowledge something that it
> has delivered to the user?
AFAIK, the rules of ack coalescing do not interact with the send
On 11/23/2015 4:00 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote:
The receive completion can be safely assumed to indicate transmit
completion over a reliable connection unless your peer has gone
completely bonkers and is replying to a
On 11/23/2015 7:00 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote:
The receive completion can be safely assumed to indicate transmit
completion over a reliable connection unless your peer has gone
completely bonkers and is replying to a
On Mon, Nov 23, 2015 at 07:34:53PM -0500, Tom Talpey wrote:
> Been there, seen that. Bluescreened on it, mysteriously.
Yes, me too :(
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at
On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote:
>The receive completion can be safely assumed to indicate transmit
>completion over a reliable connection unless your peer has gone
>completely bonkers and is replying to a command that it did not
>receive.
Perhaps
On Mon, Nov 23, 2015 at 02:33:05PM -0800, Bart Van Assche wrote:
> On 11/23/2015 02:18 PM, Jason Gunthorpe wrote:
> >On Mon, Nov 23, 2015 at 01:54:05PM -0800, Bart Van Assche wrote:
> >What I don't see is how SRP handles things when the
> >sendq fills up, ie the case where __srp_get_tx_iu() ==
On 11/23/2015 02:18 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 01:54:05PM -0800, Bart Van Assche wrote:
What I don't see is how SRP handles things when the
sendq fills up, ie the case where __srp_get_tx_iu() == NULL. It looks
like the driver starts to panic and generates printks. I can't
On Mon, Nov 23, 2015 at 01:54:05PM -0800, Bart Van Assche wrote:
> Not really ... Please have a look at the SRP initiator source code. What the
> SRP initiator does is to poll the send queue before sending a new
> SCSI
I see that. What I don't see is how SRP handles things when the
sendq fills
On 11/23/2015 01:28 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 01:04:25PM -0800, Bart Van Assche wrote:
Considerable time ago the send queue in the SRP initiator driver was
modified from signaled to non-signaled to reduce the number of interrupts
triggered by the SRP initiator driver.
On Mon, Nov 23, 2015 at 01:04:25PM -0800, Bart Van Assche wrote:
> Considerable time ago the send queue in the SRP initiator driver was
> modified from signaled to non-signaled to reduce the number of interrupts
> triggered by the SRP initiator driver. The SRP initiator driver polls the
> send
On 11/23/2015 12:37 PM, Jason Gunthorpe wrote:
On Sat, Nov 14, 2015 at 08:13:44AM +0100, Christoph Hellwig wrote:
On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote:
Looking at that thread and then at the patch a bit more..
+void ib_process_cq_direct(struct ib_cq *cq)
[..]
+
On Mon, Nov 23, 2015 at 01:01:36PM -0700, Jason Gunthorpe wrote:
> Okay, having now read the whole thing, I think I see the flow now. I don't
> see any holes in the above, other than it is doing a bit more work
> than it needs in some edges cases because it doesn't know if the CQ is
> actually
On Sat, Nov 14, 2015 at 08:13:44AM +0100, Christoph Hellwig wrote:
> On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote:
> > Looking at that thread and then at the patch a bit more..
> >
> > +void ib_process_cq_direct(struct ib_cq *cq)
> > [..]
> > + __ib_process_cq(cq, INT_MAX);
>
On Sat, Nov 14, 2015 at 08:08:49AM +0100, Christoph Hellwig wrote:
> On Fri, Nov 13, 2015 at 11:25:13AM -0700, Jason Gunthorpe wrote:
> > For instance, like this, not fulling draining the cq and then doing:
> >
> > > + completed = __ib_process_cq(cq, budget);
> > > + if (completed < budget) {
> >
On Sat, Nov 14, 2015 at 08:08:49AM +0100, Christoph Hellwig wrote:
> On Fri, Nov 13, 2015 at 11:25:13AM -0700, Jason Gunthorpe wrote:
> > For instance, like this, not fulling draining the cq and then doing:
> >
> > > + completed = __ib_process_cq(cq, budget);
> > > + if (completed < budget) {
> >
On Sat, Nov 14, 2015 at 08:13:44AM +0100, Christoph Hellwig wrote:
> On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote:
> > Looking at that thread and then at the patch a bit more..
> >
> > +void ib_process_cq_direct(struct ib_cq *cq)
> > [..]
> > + __ib_process_cq(cq, INT_MAX);
>
On 11/23/2015 02:18 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 01:54:05PM -0800, Bart Van Assche wrote:
What I don't see is how SRP handles things when the
sendq fills up, ie the case where __srp_get_tx_iu() == NULL. It looks
like the driver starts to panic and generates printks. I can't
On Mon, Nov 23, 2015 at 01:54:05PM -0800, Bart Van Assche wrote:
> Not really ... Please have a look at the SRP initiator source code. What the
> SRP initiator does is to poll the send queue before sending a new
> SCSI
I see that. What I don't see is how SRP handles things when the
sendq fills
On Mon, Nov 23, 2015 at 02:33:05PM -0800, Bart Van Assche wrote:
> On 11/23/2015 02:18 PM, Jason Gunthorpe wrote:
> >On Mon, Nov 23, 2015 at 01:54:05PM -0800, Bart Van Assche wrote:
> >What I don't see is how SRP handles things when the
> >sendq fills up, ie the case where __srp_get_tx_iu() ==
On 11/23/2015 01:28 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 01:04:25PM -0800, Bart Van Assche wrote:
Considerable time ago the send queue in the SRP initiator driver was
modified from signaled to non-signaled to reduce the number of interrupts
triggered by the SRP initiator driver.
On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote:
> Is it possible for an IB HCA to transmit a response on a QP and not
> in that packet or a previous packet acknowledge something that it
> has delivered to the user?
AFAIK, the rules of ack coalescing do not interact with the send
On Mon, Nov 23, 2015 at 07:34:53PM -0500, Tom Talpey wrote:
> Been there, seen that. Bluescreened on it, mysteriously.
Yes, me too :(
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at
On 11/23/2015 7:00 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote:
The receive completion can be safely assumed to indicate transmit
completion over a reliable connection unless your peer has gone
completely bonkers and is replying to a
On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote:
>The receive completion can be safely assumed to indicate transmit
>completion over a reliable connection unless your peer has gone
>completely bonkers and is replying to a command that it did not
>receive.
Perhaps
On 11/23/2015 4:00 PM, Jason Gunthorpe wrote:
On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote:
The receive completion can be safely assumed to indicate transmit
completion over a reliable connection unless your peer has gone
completely bonkers and is replying to a
On Mon, Nov 23, 2015 at 01:01:36PM -0700, Jason Gunthorpe wrote:
> Okay, having now read the whole thing, I think I see the flow now. I don't
> see any holes in the above, other than it is doing a bit more work
> than it needs in some edges cases because it doesn't know if the CQ is
> actually
On 11/23/2015 12:37 PM, Jason Gunthorpe wrote:
On Sat, Nov 14, 2015 at 08:13:44AM +0100, Christoph Hellwig wrote:
On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote:
Looking at that thread and then at the patch a bit more..
+void ib_process_cq_direct(struct ib_cq *cq)
[..]
+
On Mon, Nov 23, 2015 at 01:04:25PM -0800, Bart Van Assche wrote:
> Considerable time ago the send queue in the SRP initiator driver was
> modified from signaled to non-signaled to reduce the number of interrupts
> triggered by the SRP initiator driver. The SRP initiator driver polls the
> send
On 11/22/15 06:57, Sagi Grimberg wrote:
I think that bart wants to allow the caller to select cpu affinity
per CQ. In this case ib_alloc_cq in workqueue mode would need to
accept a affinity_hint from the caller (default to wild-card
WORK_CPU_UNBOUND).
Hmm, true. How would be set that hint
I think that bart wants to allow the caller to select cpu affinity
per CQ. In this case ib_alloc_cq in workqueue mode would need to
accept a affinity_hint from the caller (default to wild-card
WORK_CPU_UNBOUND).
Hmm, true. How would be set that hint from userspace? I'd really prefer
to see
On Sun, Nov 22, 2015 at 12:36:00PM +0200, Sagi Grimberg wrote:
>
>> Wouldn't it be a better idea to set the WQ_SYSFS interface and use
>> the standard sysfs interface for specifying cpumasks or node affinity?
>
> I think that bart wants to allow the caller to select cpu affinity
> per CQ. In this
Wouldn't it be a better idea to set the WQ_SYSFS interface and use
the standard sysfs interface for specifying cpumasks or node affinity?
I think that bart wants to allow the caller to select cpu affinity
per CQ. In this case ib_alloc_cq in workqueue mode would need to
accept a affinity_hint
On Sun, Nov 22, 2015 at 11:51:13AM +0200, Sagi Grimberg wrote:
>
>> Hello Christoph,
>>
>> The comment about locality in the above quote is interesting. How about
>> modifying patch 2/9 as indicated below ? The modification below does not
>> change the behavior of this patch if ib_cq.w.cpu is not
Hello Christoph,
The comment about locality in the above quote is interesting. How about
modifying patch 2/9 as indicated below ? The modification below does not
change the behavior of this patch if ib_cq.w.cpu is not modified. And it
allows users who care about locality and who want to skip
On Sun, Nov 22, 2015 at 12:36:00PM +0200, Sagi Grimberg wrote:
>
>> Wouldn't it be a better idea to set the WQ_SYSFS interface and use
>> the standard sysfs interface for specifying cpumasks or node affinity?
>
> I think that bart wants to allow the caller to select cpu affinity
> per CQ. In this
I think that bart wants to allow the caller to select cpu affinity
per CQ. In this case ib_alloc_cq in workqueue mode would need to
accept a affinity_hint from the caller (default to wild-card
WORK_CPU_UNBOUND).
Hmm, true. How would be set that hint from userspace? I'd really prefer
to see
Wouldn't it be a better idea to set the WQ_SYSFS interface and use
the standard sysfs interface for specifying cpumasks or node affinity?
I think that bart wants to allow the caller to select cpu affinity
per CQ. In this case ib_alloc_cq in workqueue mode would need to
accept a affinity_hint
Hello Christoph,
The comment about locality in the above quote is interesting. How about
modifying patch 2/9 as indicated below ? The modification below does not
change the behavior of this patch if ib_cq.w.cpu is not modified. And it
allows users who care about locality and who want to skip
On Sun, Nov 22, 2015 at 11:51:13AM +0200, Sagi Grimberg wrote:
>
>> Hello Christoph,
>>
>> The comment about locality in the above quote is interesting. How about
>> modifying patch 2/9 as indicated below ? The modification below does not
>> change the behavior of this patch if ib_cq.w.cpu is not
On 11/22/15 06:57, Sagi Grimberg wrote:
I think that bart wants to allow the caller to select cpu affinity
per CQ. In this case ib_alloc_cq in workqueue mode would need to
accept a affinity_hint from the caller (default to wild-card
WORK_CPU_UNBOUND).
Hmm, true. How would be set that hint
On 11/20/2015 02:16 AM, Christoph Hellwig wrote:
> On Wed, Nov 18, 2015 at 10:20:14AM -0800, Bart Van Assche wrote:
>> Are you perhaps referring to the sysfs CPU mask that allows to control
>> workqueue affinity ?
>
> I think he is referring to the defintion of WQ_UNBOUND:
>
>WQ_UNBOUND
>
>
On Wed, Nov 18, 2015 at 10:20:14AM -0800, Bart Van Assche wrote:
> Are you perhaps referring to the sysfs CPU mask that allows to control
> workqueue affinity ?
I think he is referring to the defintion of WQ_UNBOUND:
WQ_UNBOUND
Work items queued to an unbound wq are served by the
On Wed, Nov 18, 2015 at 10:20:14AM -0800, Bart Van Assche wrote:
> Are you perhaps referring to the sysfs CPU mask that allows to control
> workqueue affinity ?
I think he is referring to the defintion of WQ_UNBOUND:
WQ_UNBOUND
Work items queued to an unbound wq are served by the
On 11/20/2015 02:16 AM, Christoph Hellwig wrote:
> On Wed, Nov 18, 2015 at 10:20:14AM -0800, Bart Van Assche wrote:
>> Are you perhaps referring to the sysfs CPU mask that allows to control
>> workqueue affinity ?
>
> I think he is referring to the defintion of WQ_UNBOUND:
>
>WQ_UNBOUND
>
>
On 11/17/2015 11:55 PM, Sagi Grimberg wrote:
+static void ib_cq_poll_work(struct work_struct *work)
+{
+struct ib_cq *cq = container_of(work, struct ib_cq, work);
+int completed;
+
+completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
+if (completed >=
On Tue, Nov 17, 2015 at 09:52:58AM -0800, Bart Van Assche wrote:
> On 11/13/2015 05:46 AM, Christoph Hellwig wrote:
>> + * context and does not ask from completion interrupts from the HCA.
>
> Should this perhaps be changed into "for" ?
Yes.
>
>> + */
>> +void
On 11/17/2015 11:55 PM, Sagi Grimberg wrote:
+static void ib_cq_poll_work(struct work_struct *work)
+{
+struct ib_cq *cq = container_of(work, struct ib_cq, work);
+int completed;
+
+completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
+if (completed >=
On Tue, Nov 17, 2015 at 09:52:58AM -0800, Bart Van Assche wrote:
> On 11/13/2015 05:46 AM, Christoph Hellwig wrote:
>> + * context and does not ask from completion interrupts from the HCA.
>
> Should this perhaps be changed into "for" ?
Yes.
>
>> + */
>> +void
Hi Bart,
+ */
+void ib_process_cq_direct(struct ib_cq *cq)
+{
+WARN_ON_ONCE(cq->poll_ctx != IB_POLL_DIRECT);
+
+__ib_process_cq(cq, INT_MAX);
+}
+EXPORT_SYMBOL(ib_process_cq_direct);
My proposal is to drop this function and to export __ib_process_cq()
instead (with or without
On 11/13/2015 05:46 AM, Christoph Hellwig wrote:
+ * context and does not ask from completion interrupts from the HCA.
Should this perhaps be changed into "for" ?
+ */
+void ib_process_cq_direct(struct ib_cq *cq)
+{
+ WARN_ON_ONCE(cq->poll_ctx !=
On 11/13/2015 05:46 AM, Christoph Hellwig wrote:
+ * context and does not ask from completion interrupts from the HCA.
Should this perhaps be changed into "for" ?
+ */
+void ib_process_cq_direct(struct ib_cq *cq)
+{
+ WARN_ON_ONCE(cq->poll_ctx !=
Hi Bart,
+ */
+void ib_process_cq_direct(struct ib_cq *cq)
+{
+WARN_ON_ONCE(cq->poll_ctx != IB_POLL_DIRECT);
+
+__ib_process_cq(cq, INT_MAX);
+}
+EXPORT_SYMBOL(ib_process_cq_direct);
My proposal is to drop this function and to export __ib_process_cq()
instead (with or without
On 15/11/2015 14:55, Christoph Hellwig wrote:
On Sun, Nov 15, 2015 at 11:40:02AM +0200, Sagi Grimberg wrote:
I doubt INT_MAX is useful as a budget in any use-case. it can easily
hog the CPU. If the consumer is given access to poll a CQ, it must be
able to provide some way to budget it. Why
On Sun, Nov 15, 2015 at 11:40:02AM +0200, Sagi Grimberg wrote:
> I doubt INT_MAX is useful as a budget in any use-case. it can easily
> hog the CPU. If the consumer is given access to poll a CQ, it must be
> able to provide some way to budget it. Why not expose a budget argument
> to the consumer?
+/**
+ * ib_process_direct_cq - process a CQ in caller context
+ * @cq:CQ to process
+ *
+ * This function is used to process all outstanding CQ entries on a
+ * %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different
+ * context and does not ask from completion
+/**
+ * ib_process_direct_cq - process a CQ in caller context
+ * @cq:CQ to process
+ *
+ * This function is used to process all outstanding CQ entries on a
+ * %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different
+ * context and does not ask from completion
On 15/11/2015 14:55, Christoph Hellwig wrote:
On Sun, Nov 15, 2015 at 11:40:02AM +0200, Sagi Grimberg wrote:
I doubt INT_MAX is useful as a budget in any use-case. it can easily
hog the CPU. If the consumer is given access to poll a CQ, it must be
able to provide some way to budget it. Why
On Sun, Nov 15, 2015 at 11:40:02AM +0200, Sagi Grimberg wrote:
> I doubt INT_MAX is useful as a budget in any use-case. it can easily
> hog the CPU. If the consumer is given access to poll a CQ, it must be
> able to provide some way to budget it. Why not expose a budget argument
> to the consumer?
On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote:
> Looking at that thread and then at the patch a bit more..
>
> +void ib_process_cq_direct(struct ib_cq *cq)
> [..]
> + __ib_process_cq(cq, INT_MAX);
>
> INT_MAX is not enough, it needs to loop.
> This is missing a
On Fri, Nov 13, 2015 at 11:25:13AM -0700, Jason Gunthorpe wrote:
> For instance, like this, not fulling draining the cq and then doing:
>
> > + completed = __ib_process_cq(cq, budget);
> > + if (completed < budget) {
> > + irq_poll_complete(>iop);
> > + if
On Fri, Nov 13, 2015 at 11:57:56AM -0800, Bart Van Assche wrote:
> I think this is the conversation you are referring to: "About a shortcoming
> of the verbs API" (http://thread.gmane.org/gmane.linux.drivers.rdma/5028).
> That conversation occurred five years ago, which means that you have an
>
On 11/13/2015 10:25 AM, Jason Gunthorpe wrote:
On Fri, Nov 13, 2015 at 02:46:43PM +0100, Christoph Hellwig wrote:
This adds an abstraction that allows ULP to simply pass a completion
object and completion callback with each submitted WR and let the RDMA
core handle the nitty gritty details of
On Fri, Nov 13, 2015 at 02:46:43PM +0100, Christoph Hellwig wrote:
> This adds an abstraction that allows ULP to simply pass a completion
> object and completion callback with each submitted WR and let the RDMA
> core handle the nitty gritty details of how to handle completion
> interrupts and
This adds an abstraction that allows ULP to simply pass a completion
object and completion callback with each submitted WR and let the RDMA
core handle the nitty gritty details of how to handle completion
interrupts and poll the CQ.
In detail there is a new ib_cqe structure which just contains
On 11/13/2015 10:25 AM, Jason Gunthorpe wrote:
On Fri, Nov 13, 2015 at 02:46:43PM +0100, Christoph Hellwig wrote:
This adds an abstraction that allows ULP to simply pass a completion
object and completion callback with each submitted WR and let the RDMA
core handle the nitty gritty details of
On Fri, Nov 13, 2015 at 02:46:43PM +0100, Christoph Hellwig wrote:
> This adds an abstraction that allows ULP to simply pass a completion
> object and completion callback with each submitted WR and let the RDMA
> core handle the nitty gritty details of how to handle completion
> interrupts and
On Fri, Nov 13, 2015 at 11:57:56AM -0800, Bart Van Assche wrote:
> I think this is the conversation you are referring to: "About a shortcoming
> of the verbs API" (http://thread.gmane.org/gmane.linux.drivers.rdma/5028).
> That conversation occurred five years ago, which means that you have an
>
On Fri, Nov 13, 2015 at 11:25:13AM -0700, Jason Gunthorpe wrote:
> For instance, like this, not fulling draining the cq and then doing:
>
> > + completed = __ib_process_cq(cq, budget);
> > + if (completed < budget) {
> > + irq_poll_complete(>iop);
> > + if
On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote:
> Looking at that thread and then at the patch a bit more..
>
> +void ib_process_cq_direct(struct ib_cq *cq)
> [..]
> + __ib_process_cq(cq, INT_MAX);
>
> INT_MAX is not enough, it needs to loop.
> This is missing a
This adds an abstraction that allows ULP to simply pass a completion
object and completion callback with each submitted WR and let the RDMA
core handle the nitty gritty details of how to handle completion
interrupts and poll the CQ.
In detail there is a new ib_cqe structure which just contains
76 matches
Mail list logo