Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 10/04/17 02:29 AM, Sagi Grimberg wrote: > What you are saying is surprising to me. The switch needs to preserve > ordering across different switch ports ?? > You are suggesting that there is a *switch-wide* state that tracks > MemRds never pass MemWrs across all the switch ports? That is a

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 10/04/17 02:29 AM, Sagi Grimberg wrote: > What you are saying is surprising to me. The switch needs to preserve > ordering across different switch ports ?? > You are suggesting that there is a *switch-wide* state that tracks > MemRds never pass MemWrs across all the switch ports? That is a

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

Sagi As long as legA, legB and the RC are all connected to the same switch then ordering will be preserved (I think many other topologies also work). Here is how it would work for the problem case you are concerned about (which is a read from the NVMe drive). 1. Disk device DMAs out the

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

Sagi As long as legA, legB and the RC are all connected to the same switch then ordering will be preserved (I think many other topologies also work). Here is how it would work for the problem case you are concerned about (which is a read from the NVMe drive). 1. Disk device DMAs out the

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 2017-04-06, 6:33 AM, "Sagi Grimberg" wrote: > Say it's connected via 2 legs, the bar is accessed from leg A and the > data from the disk comes via leg B. In this case, the data is heading > towards the p2p device via leg B (might be congested), the completion > goes directly

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 2017-04-06, 6:33 AM, "Sagi Grimberg" wrote: > Say it's connected via 2 legs, the bar is accessed from leg A and the > data from the disk comes via leg B. In this case, the data is heading > towards the p2p device via leg B (might be congested), the completion > goes directly to the RC, and

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On Thu, Apr 06, 2017 at 08:33:38AM +0300, Sagi Grimberg wrote: > > >>Note that the nvme completion queues are still on the host memory, so > >>this means we have lost the ordering between data and completions as > >>they go to different pcie targets. > > > >Hmm, in this simple up/down case with a

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On Thu, Apr 06, 2017 at 08:33:38AM +0300, Sagi Grimberg wrote: > > >>Note that the nvme completion queues are still on the host memory, so > >>this means we have lost the ordering between data and completions as > >>they go to different pcie targets. > > > >Hmm, in this simple up/down case with a

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 05/04/17 11:33 PM, Sagi Grimberg wrote: > >>> Note that the nvme completion queues are still on the host memory, so >>> this means we have lost the ordering between data and completions as >>> they go to different pcie targets. >> >> Hmm, in this simple up/down case with a switch, I think it

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 05/04/17 11:33 PM, Sagi Grimberg wrote: > >>> Note that the nvme completion queues are still on the host memory, so >>> this means we have lost the ordering between data and completions as >>> they go to different pcie targets. >> >> Hmm, in this simple up/down case with a switch, I think it

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

Note that the nvme completion queues are still on the host memory, so this means we have lost the ordering between data and completions as they go to different pcie targets. Hmm, in this simple up/down case with a switch, I think it might actually be OK. Transactions might not complete at

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

Note that the nvme completion queues are still on the host memory, so this means we have lost the ordering between data and completions as they go to different pcie targets. Hmm, in this simple up/down case with a switch, I think it might actually be OK. Transactions might not complete at

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 04/04/17 04:59 AM, Sagi Grimberg wrote: > We can never ever get here from an IO command, and that is a good thing > because it would have been broken if we did, regardless of what copy > method we use... Yes, I changed this mostly for admin commands. I did notice connect commands do end up

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On 04/04/17 04:59 AM, Sagi Grimberg wrote: > We can never ever get here from an IO command, and that is a good thing > because it would have been broken if we did, regardless of what copy > method we use... Yes, I changed this mostly for admin commands. I did notice connect commands do end up

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On Tue, Apr 04, 2017 at 01:59:26PM +0300, Sagi Grimberg wrote: > Note that the nvme completion queues are still on the host memory, so > this means we have lost the ordering between data and completions as > they go to different pcie targets. Hmm, in this simple up/down case with a switch, I

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

On Tue, Apr 04, 2017 at 01:59:26PM +0300, Sagi Grimberg wrote: > Note that the nvme completion queues are still on the host memory, so > this means we have lost the ordering between data and completions as > they go to different pcie targets. Hmm, in this simple up/down case with a switch, I

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

u16 nvmet_copy_to_sgl(struct nvmet_req *req, off_t off, const void *buf, size_t len) { - if (sg_pcopy_from_buffer(req->sg, req->sg_cnt, buf, len, off) != len) + bool iomem = req->p2pmem; + size_t ret; + + ret = sg_copy_buffer(req->sg, req->sg_cnt, (void

Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

u16 nvmet_copy_to_sgl(struct nvmet_req *req, off_t off, const void *buf, size_t len) { - if (sg_pcopy_from_buffer(req->sg, req->sg_cnt, buf, len, off) != len) + bool iomem = req->p2pmem; + size_t ret; + + ret = sg_copy_buffer(req->sg, req->sg_cnt, (void

[RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

p2pmem will always be iomem so if we ever access it, we should be using the correct methods to read and write to it. Signed-off-by: Logan Gunthorpe Signed-off-by: Stephen Bates Signed-off-by: Steve Wise ---

[RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem

p2pmem will always be iomem so if we ever access it, we should be using the correct methods to read and write to it. Signed-off-by: Logan Gunthorpe Signed-off-by: Stephen Bates Signed-off-by: Steve Wise --- drivers/nvme/target/core.c| 18 --