Re: simple example of pci driver with dma

[Yan Vugenfirer]

Hi Shaked,

In the prob function, before you are trying to do any DMA operations with your 
device, you should call pci_enable_device and then pci_set_master. Also you 
might need to map the resources of your device.
Check PCI driver documentation: 
https://lxr.missinglinkelectronics.com/linux/Documentation/PCI/pci.rst#L199

Best regards,
Yan.

> On 29 Oct 2020, at 10:32 PM, Shaked Matzner  wrote:
> 
> Hey Peter,
> Currently I have this test in the driver which allocates data, assign it to 
> default value config source as the BASE address of dma, dest as the physical 
> address which I got from dma_alloc_coherent, set the count and assign the 
> command to raise interrupt and read,the piece of code used is something like 
> this(this a test performed from the probe function of the driver)
> vaddr_to = 
> dma_alloc_coherent(&(dev->dev), 4, _handle_to, GFP_ATOMIC |GFP_KERNEL);
> *((volatile int*)vaddr_to) = 0xff;
> test->vaddr_to = vaddr_to;
> dev_info(&(dev->dev), 
> "vaddr_to = %px\n", vaddr_to);
> dev_info(&(dev->dev), 
> "dma_handle_to = %llx\n", (unsigned long long)dma_handle_to);
> iowrite32(DMA_BASE, mmio + 
> IO_DMA_SRC);
> iowrite32((u32)dma_handle_to, 
> mmio + IO_DMA_DST);
> iowrite32(SIZE, mmio + 
> IO_DMA_CNT);
>iowrite32(DMA_CMD | 
> DMA_FROM_DEV | DMA_IRQ, mmio + IO_DMA_CMD);
> Where and when should the pci_set_master hould be called?
> Thanks,
>  Shaked Matzner
>  
> From: Peter Maydell  
> Sent: Thursday, October 29, 2020 5:46 PM
> To: Shaked Matzner 
> Cc: qemu-devel@nongnu.org
> Subject: Re: simple example of pci driver with dma
>  
> 
> IRONSCALES couldn't recognize this email as this is the first time you 
> received an email from this sender peter.mayd...@linaro.org
>  
> [EXTERNAL]
> 
> On Thu, 29 Oct 2020 at 14:59, Shaked Matzner wrote:
> > however the value I get is still 255(0xff) and not 18(0x12) probably I've 
> > missed something but when the interrupt is called the transfer to the RAM 
> > address should be completed, however it seems like the dma_write_buffer 
> > function from the device does not perform any transfer. What Am I missing?
> 
> The usual mistake is forgetting in the guest code to program the
> PCI device to enable bus mastering by setting the Bus Master bit
> in the Command register in the PCI config space registers for
> the device. Unless you do that then all DMA attempts will fail
> (same as on real h/w). In the Linux kernel the function for this
> is pci_set_master(), I think.
> 
> thanks
> -- PMM 
> 
> The contents of this email message and any attachments are intended solely 
> for the addressee(s) and may contain confidential and/or privileged 
> information and may be legally protected from disclosure. If you are not the 
> intended recipient of this message or their agent, or if this message has 
> been addressed to you in error, please immediately alert the sender by reply 
> email and then delete this message and any attachments. If you are not the 
> intended recipient, you are hereby notified that any use, dissemination, 
> copying, or storage of this message or its attachments is strictly 
> prohibited. 
> 



--
Daynix Computing LTD
Yan Vugenfirer, CEO
Email: y...@daynix.com
Phone (Israel): +972-54-4758084
Phone (USA): +1-7204776716
Phone (UK): +44-2070482938
Web: www.daynix.com

RE: simple example of pci driver with dma

[Shaked Matzner]

Hey Peter,
Currently I have this test in the driver which allocates data, assign it to 
default value config source as the BASE address of dma, dest as the physical 
address which I got from dma_alloc_coherent, set the count and assign the 
command to raise interrupt and read,the piece of code used is something like 
this(this a test performed from the probe function of the driver)
vaddr_to = 
dma_alloc_coherent(&(dev->dev), 4, _handle_to, GFP_ATOMIC |GFP_KERNEL);
*((volatile int*)vaddr_to) = 0xff;
test->vaddr_to = vaddr_to;
dev_info(&(dev->dev), "vaddr_to 
= %px\n", vaddr_to);
dev_info(&(dev->dev), 
"dma_handle_to = %llx\n", (unsigned long long)dma_handle_to);
iowrite32(DMA_BASE, mmio + 
IO_DMA_SRC);
iowrite32((u32)dma_handle_to, 
mmio + IO_DMA_DST);
iowrite32(SIZE, mmio + 
IO_DMA_CNT);
   iowrite32(DMA_CMD | DMA_FROM_DEV 
| DMA_IRQ, mmio + IO_DMA_CMD);
Where and when should the pci_set_master hould be called?
Thanks,
 Shaked Matzner

From: Peter Maydell 
Sent: Thursday, October 29, 2020 5:46 PM
To: Shaked Matzner 
Cc: qemu-devel@nongnu.org
Subject: Re: simple example of pci driver with dma

[https://s3.amazonaws.com/staticmediafiles/media/sights/iron-icon-color.png]
IRONSCALES couldn't recognize this email as this is the first time you received 
an email from this sender 
peter.mayd...@linaro.org<mailto:peter.mayd...@linaro.org>

[EXTERNAL]

On Thu, 29 Oct 2020 at 14:59, Shaked Matzner wrote:
> however the value I get is still 255(0xff) and not 18(0x12) probably I've 
> missed something but when the interrupt is called the transfer to the RAM 
> address should be completed, however it seems like the dma_write_buffer 
> function from the device does not perform any transfer. What Am I missing?

The usual mistake is forgetting in the guest code to program the
PCI device to enable bus mastering by setting the Bus Master bit
in the Command register in the PCI config space registers for
the device. Unless you do that then all DMA attempts will fail
(same as on real h/w). In the Linux kernel the function for this
is pci_set_master(), I think.

thanks
-- PMM


The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or privileged information and 
may be legally protected from disclosure. If you are not the intended recipient 
of this message or their agent, or if this message has been addressed to you in 
error, please immediately alert the sender by reply email and then delete this 
message and any attachments. If you are not the intended recipient, you are 
hereby notified that any use, dissemination, copying, or storage of this 
message or its attachments is strictly prohibited.

Re: simple example of pci driver with dma

[Peter Maydell]

On Thu, 29 Oct 2020 at 14:59, Shaked Matzner  wrote:
> however the value I get is still 255(0xff) and not 18(0x12) probably I've 
> missed something but when the interrupt is called the transfer to the RAM 
> address should be completed, however it seems like the dma_write_buffer 
> function from the device does not perform any transfer. What Am I missing?

The usual mistake is forgetting in the guest code to program the
PCI device to enable bus mastering by setting the Bus Master bit
in the Command register in the PCI config space registers for
the device. Unless you do that then all DMA attempts will fail
(same as on real h/w). In the Linux kernel the function for this
is pci_set_master(), I think.

thanks
-- PMM

simple example of pci driver with dma

[Shaked Matzner]

Hey All,

So I'm trying to learn how pci devices and drivers work using the edu device 
and an educational driver, and It seems like the pci_dma_write function fails 
to actually write the information from the the dma buffer into the allocated 
address in the ram. Here is How I testsed it: 1.I've initialized the first byte 
of the dma buff to be 0x12 in the realize function of the edu:

 edu->dma_buf[0] = 0x12;

In the driver I've defined a struct save both the major and the destination 
address allocated through the dma cohherent:

 struct test {

  int major;

  void *vaddr_to;

};

allocated it on the probe function and commanded the device to read 4 bytes:

 dev_info(&(dev->dev),"Hello from Shaked\n");

 dma_addr_t dma_handle_from, dma_handle_to;

 void *vaddr_from, *vaddr_to;

 enum { SIZE = 4 };



 /* device -> RAM. */

   vaddr_to = dma_alloc_coherent(&(dev->dev), 4, _handle_to, GFP_ATOMIC);

   *((volatile int*)vaddr_to) = 0xff;

test->vaddr_to = vaddr_to;

dev_info(&(dev->dev), "vaddr_to = %px\n", vaddr_to);

dev_info(&(dev->dev), "dma_handle_to = %llx\n", (unsigned long 
long)dma_handle_to);

/*write source, dest, number of bytes to transfer and activate the dma 
timer)/*

iowrite32(DMA_BASE, mmio + IO_DMA_SRC);

iowrite32((u32)dma_handle_to, mmio + IO_DMA_DST);

iowrite32(SIZE, mmio + IO_DMA_CNT);

iowrite32(DMA_CMD | DMA_FROM_DEV | DMA_IRQ, mmio + IO_DMA_CMD);

I've changed a little bit the interrtupt and added a print to check the value I 
just read:

static irqreturn_t irq_handler(int irq, void *dev)

{

int devi;

irqreturn_t ret;

u32 irq_status;

struct test* test;



test =  (struct test *)dev;

if (test->major == major) {

irq_status = ioread32(mmio + IO_IRQ_STATUS);

pr_info("irq_handler irq = %d dev = %d irq_status = %llx\n",

irq, devi, (unsigned long long)irq_status);

/* Must do this ACK, or else the interrupts just keeps firing. */

iowrite32(irq_status, mmio + IO_IRQ_ACK);

pr_info("*vaddr_to new_value = %u\n", (*((u8*)test->vaddr_to)));



ret = IRQ_HANDLED;

} else {

ret = IRQ_NONE;

}

return ret;

}

however the value I get is still 255(0xff) and not 18(0x12) probably I've 
missed something but when the interrupt is called the transfer to the RAM 
address should be completed, however it seems like the dma_write_buffer 
function from the device does not perform any transfer. What Am I missing?

when I debugged the dma_coherent_alloc I saw it uses direct acccess(without 
iommu), however when the dma_pci_write is called it tries to access the memory 
via iommu and get's an error code(MEMTX_DECODE_ERROR) from memory.c 's 
memory_region_dispatch_write. Since memory_region_access_valid returns false on 
it's first condition(also under memory.c

I am now in a deadlock, since even this simple example  doesn't seem to work, 
any idea what happens, or any suggestions for an alternative simple device with 
a simple driver which I can learn from will be great.

Technical details: edu_device: 
https://github.com/qemu/qemu/blob/master/hw/misc/edu.c original edu_driver: 
https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/qemu_edu.c
 I run this on an x86-64 qemu machine with the following configuration:

$repo_loc/build/qemu-system-x86_64 \

-no-kvm \

-kernel $repo_loc/linux-5.8.5/arch/x86/boot/bzImage \

-boot c -m 2049M \

-hda $repo_loc/buildroot-2020.02.6/output/images/rootfs.ext4 \

-append "root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr" \

-serial stdio -display none \

-virtfs 
local,path=$repo_loc/shared,mount_tag=host0,security_model=passthrough,id=host0 
\

-device edu

(where $repo_location is the path to my build dir)


Thanks,
   Shaked Matzner



The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or privileged information and 
may be legally protected from disclosure. If you are not the intended recipient 
of this message or their agent, or if this message has been addressed to you in 
error, please immediately alert the sender by reply email and then delete this 
message and any attachments. If you are not the intended recipient, you are 
hereby notified that any use, dissemination, copying, or storage of this 
message or its attachments is strictly prohibited.