Hi, > Subject: Re: [dpdk-dev] DPDK doesn't work with iommu=pt > > iommu=pt effectively disables iommu for the kernel and iommu is > enabled only for KVM. > http://lwn.net/Articles/329174/
thanks for pointing that. Okay, I think DPDK cannot handle IOMMU because of no kernel code in DPDK application. And now, I think "iommu=pt" doesn't work correctly DMA on host PMD causes DMAR fault which means IOMMU catches a wrong operation. Will dig around "iommu=pt". > > Basically unless you have KVM running you can remove both lines for > the same effect. > On the other hand if you do have KVM and you do want iommu=on You can > remove the iommu=pt for the same performance because AFAIK unlike the > kernel drivers DPDK doesn't dma_map and dma_unman each and every > ingress/egress packet (Please correct me if I'm wrong), and will not > suffer any performance penalties. I also tried "iommu=on", but it didn't fix the issue. I saw the same error messages in kernel. [ 46.978097] dmar: DRHD: handling fault status reg 2 [ 46.978120] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr aa010000 DMAR:[fault reason 02] Present bit in context entry is clear thanks, Hiroshi > > FYI. Kernel NIC drivers: > When iommu=on{,strict} the kernel network drivers will suffer a heavy > performance penalty due to regular IOVA modifications (both HW and SW > at fault here). Ixgbe and Mellanox reuse dma_mapped pages on the > receive side to avoid this penalty, but still suffer from iommu on TX. > > On Fri, Sep 26, 2014 at 5:47 PM, Choi, Sy Jong <sy.jong.choi at intel.com> > wrote: > > Hi Shimamoto-san, > > > > There are a lot of sighting relate to "DMAR:[fault reason 06] PTE Read > > access is not set" > > https://www.mail-archive.com/kvm at vger.kernel.org/msg106573.html > > > > This might be related to IOMMU, and kernel code. > > > > Here is what we know :- > > 1) Disabling VT-d in bios also removed the symptom > > 2) Switch to another OS distribution also removed the symptom > > 3) even different HW we will not see the symptom. In my case, switch from > > Engineering board to EPSD board. > > > > Regards, > > Choi, Sy Jong > > Platform Application Engineer > > > > > > -----Original Message----- > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Hiroshi Shimamoto > > Sent: Friday, September 26, 2014 5:14 PM > > To: dev at dpdk.org > > Cc: Hayato Momma > > Subject: [dpdk-dev] DPDK doesn't work with iommu=pt > > > > I encountered an issue that DPDK doesn't work with "iommu=pt intel_iommu=on" > > on HP ProLiant DL380p Gen8 server. I'm using the following environment; > > > > HW: ProLiant DL380p Gen8 > > CPU: E5-2697 v2 > > OS: RHEL7 > > kernel: kernel-3.10.0-123 and the latest kernel 3.17-rc6+ > > DPDK: v1.7.1-53-gce5abac > > NIC: 82599ES > > > > When boot with "iommu=pt intel_iommu=on", I got the below message and no > > packets are handled. > > > > [ 120.809611] dmar: DRHD: handling fault status reg 2 > > [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr > > aa010000 > > DMAR:[fault reason 02] Present bit in context entry is clear > > > > How to reproduce; > > just run testpmd > > # ./testpmd -c 0xf -n 4 -- -i > > > > Configuring Port 0 (socket 0) > > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7ffff54eafc0 > > hw_ring=0x7ffff4200000 dma_addr=0xaa000000 > > PMD: ixgbe_dev_tx_queue_setup(): Using full-featured tx code path > > PMD: ixgbe_dev_tx_queue_setup(): - txq_flags = 0 [IXGBE_SIMPLE_FLAGS=f01] > > PMD: ixgbe_dev_tx_queue_setup(): - tx_rs_thresh = 32 > > [RTE_PMD_IXGBE_TX_MAX_BURST=32] > > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7ffff54ea740 > > hw_ring=0x7ffff4210000 dma_addr=0xaa010000 > > PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc > > Preconditions: rxq->rx_free_thresh=0, > RTE_PMD_IXGBE_RX_MAX_BURST=32 > > PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are not > > satisfied, Scattered Rx is requested, or > RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC is not enabled (port=0, queue=0). > > PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc > > Preconditions: rxq->rx_free_thresh=0, > RTE_PMD_IXGBE_RX_MAX_BURST=32 > > > > testpmd> start > > io packet forwarding - CRC stripping disabled - packets/burst=32 > > nb forwarding cores=1 - nb forwarding ports=2 > > RX queues=1 - RX desc=128 - RX free threshold=0 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=0 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=0 - TXQ flags=0x0 > > > > > > and ping from another box to this server. > > # ping6 -I eth2 ff02::1 > > > > I got the below error message and no packet is received. > > I couldn't see any increase RX/TX count in testpmt statistics > > > > testpmd> show port stats 0 > > > > ######################## NIC statistics for port 0 > > ######################## > > RX-packets: 6 RX-missed: 0 RX-bytes: 732 > > RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 > > RX-nombuf: 0 > > TX-packets: 0 TX-errors: 0 TX-bytes: 0 > > > > ############################################################################ > > testpmd> show port stats 0 > > > > ######################## NIC statistics for port 0 > > ######################## > > RX-packets: 6 RX-missed: 0 RX-bytes: 732 > > RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 > > RX-nombuf: 0 > > TX-packets: 0 TX-errors: 0 TX-bytes: 0 > > > > ############################################################################ > > > > > > The fault addr in error message must be RX DMA descriptor > > > > error message > > [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr > > aa010000 > > > > log in testpmd > > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7ffff54ea740 > > hw_ring=0x7ffff4210000 dma_addr=0xaa010000 > > > > I think the NIC received a packet in fifo and try to put into memory with > > DMA. > > Before starting DMA, the NIC get the target address from RX descriptors in > > RDBA register. > > But accessing RX descriptors failed in IOMMU unit and reported it to the > > kernel. > > > > DMAR:[fault reason 02] Present bit in context entry is clear > > > > The error message looks there is no valid entry in IOMMU. > > > > I think the following issue is very similar, but using Ubuntu14.04 couldn't > > fix in my case. > > http://thread.gmane.org/gmane.comp.networking.dpdk.devel/2281 > > > > I tried Ubuntu14.04.1 and got the below error. > > > > [ 199.710191] dmar: DRHD: handling fault status reg 2 > > [ 199.710896] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr > > 7c24df000 > > [ 199.710896] DMAR:[fault reason 06] PTE Read access is not set > > > > Currently I could see this issue on HP ProLiant DL380p Gen8 only. > > Is there any idea? > > Has anyone noticed this issue? > > > > Note: we're thinking to use SR-IOV and DPDK app in the same box. > > The box has 2 NICs, one for SR-IOV and pass through to VM, one (no SR-IOV) > > for DPDK app in host. > > > > thanks, > > Hiroshi