Re: DMA to User-Space
On Wednesday 04 November 2009 02:37:38 Jonathan Haws wrote: All, I have what may be an unconventional question: Our application consists of data being captured by an FPGA, processed, and transferred to SDRAM. I simply give the FPGA an address of where I want it stored in SDRAM and it simply DMAs the data over and interrupts me when finished. I then take that data and store it to disk. I have code in user space that handles all of the writing to disk nicely and fast enough for my application (I am capturing data at about 35-40 Mbytes/sec). My question is this: is it possible to give a user-space pointer to the FPGA to DMA to? It seems like I would have problems with alignment, address manipulation, and a whole slew of other issues. Well, it's possible: you'll have to use zero-copy I/O. Here's an article on this subject: http://lwn.net/Articles/28548/. Have a look at drivers/scsi/st.c at sgl_get_user_pages() or drivers/infiniband/hw/ipath/ipath_user_pages.c for example of how to use that. Regards, Sergey Temerkhanov. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: DMA to User-Space
On 11/04/2009 11:50 AM, Jonathan Haws wrote: One more question about this approach: does the mmap() call prevent the kernel from using this memory for other purposes? Will the kernel be able to move this memory elsewhere? I guess what I am asking is if this memory is locked for all other purposes? You've allocated the memory in the kernel and mapped it to userspace. If the kernel uses that memory for anything else it will be visible to userspace. Chris ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
RE: DMA to User-Space
Jonathan Haws wrote:
All,
I have what may be an unconventional question:
Our application consists of data being captured by an FPGA,
processed, and transferred to SDRAM. I simply give the FPGA an
address of where I want it stored in SDRAM and it simply DMAs the
data over and interrupts me when finished. I then take that data
and store it to disk.
I have code in user space that handles all of the writing to disk
nicely and fast enough for my application (I am capturing data at
about 35-40 Mbytes/sec).
My question is this: is it possible to give a user-space pointer
to the FPGA to DMA to? It seems like I would have problems with
alignment, address manipulation, and a whole slew of other issues.
What would be the best way to accomplish something like that? I
want to handle all the disk access in user-space, but I do not want
to have to copy 40 MB/s from kernel space to user-space either.
You can maintain a DMA buffer in kernel, then mmap to user space.
And
maybe you need some handshake between FPGA and the apps to balance
input
datas with datas to disk.
I can maintain an allocated, DMA-safe buffer in kernel space if
needed. Can I simply get a user-space pointer to that buffer? What
calls are needed to translate addresses?
Use remap_pfn_range() in your kernel DMA buffer manipulation driver
.mmap() handler to export DMA buffer address to user space.
Can you provide an example for how to do that? I have an mmap routine to map
BARs that the FPGA maintains and I can access those, however when I try to map
the DMA buffer and access what is in it, the system crashes. Here is the mmap
function I have right now:
/* fpga_mmap()
*
* Description:
* The purpose of this function is to serve as a 'file_operation'
* which maps different PCI resources into the calling processes
* memory space.
*
* NOTE: The file offset are in page size; i.e.:
* offset 0 in process's mmap syscall - BAR0
* offset 4096 in process's mmap syscall - BAR1
* offset 8192 in process's mmap syscall - BAR2
* offset 12288 - Streaming DMA buffer
*
* Arguments:
* struct file *filp -- struct representing an open file
* struct vm_area_struct *vma -- struct representing memory 'segment'
*
* Returns:
* int -- indication of success or failure
*
*/
int fpga_mmap(struct file *filp, struct vm_area_struct *vma)
{
struct pci_dev *dev;
unsigned long addressToMap;
uint8_t mapType = 0; /* 0 = IO, 1 = memory */
/* Get the PCI device */
dev = (struct pci_dev*)(filp-private_data);
/* Map in the appropriate BAR based on page offset */
if (vma-vm_pgoff == FPGA_CONFIG_SPACE)
{
/* Map BAR1 (the CONFIG area) */
printk(KERN_ALERT FPGA: Mapping BAR1 (CONFIG BAR).\n);
addressToMap = pci_resource_start(dev, FPGA_CONFIG_SPACE);
printk(KERN_ALERT FPGA: PCI BAR1 (CONFIG BAR) Size -
%#08x.\n,
pci_resource_len(dev, FPGA_CONFIG_SPACE));
mapType = 0;
}
else if(vma-vm_pgoff == FPGA_TEST_SPACE)
{
/* Map BAR2 (the TEST area) */
printk(KERN_ALERT FPGA: Mapping BAR2 (TEST BAR).\n);
addressToMap = (pci_resource_start(dev, FPGA_TEST_SPACE) +
pci_resource_len(dev, FPGA_TEST_SPACE)) -
FPGA_TEST_LENGTH;
printk(KERN_ALERT FPGA: PCI BAR2 (TEST BAR) Size - %#08x.\n,
pci_resource_len(dev, FPGA_TEST_SPACE));
mapType = 0;
}
else if(vma-vm_pgoff == 3)
{
addressToMap = (unsigned long)fpga_drv.strmData[0];
mapType = 1;
}
else
{
printk(KERN_ALERT FPGA: Invalid BAR mapping specified.\n);
return ERROR;
}
/* Execute the mapping */
vma-vm_flags |= VM_IO;
vma-vm_flags |= VM_RESERVED;
vma-vm_page_prot = pgprot_noncached(vma-vm_page_prot);
printk(KERN_ALERT FPGA: vmSize - 0x%x.\n,
(unsigned int)(vma-vm_end - vma-vm_start));
if( mapType == 0 )
{
if(io_remap_pfn_range(vma, vma-vm_start, addressToMap
PAGE_SHIFT,
vma-vm_end - vma-vm_start, vma-vm_page_prot)
!= 0)
{
printk(KERN_ALERT FPGA: Failed to map BAR PCI space to
user space.\n);
return ERROR;
}
}
else
{
printk(KERN_NOTICE FPGA: Mapping stream ptr (%#08x) to user
space\n,(uint32_t)fpga_drv.strmData[0]);
printk(KERN_NOTICE FPGA: Setting strmData[0][0] to 0x37\n);
fpga_drv.strmData[0][0] = 0x37;
if(remap_pfn_range(vma, vma-vm_start, addressToMap
PAGE_SHIFT,
RE: DMA to User-Space
1. I open /dev/mem and get a file descriptor 2. I use mmap to reserve some physical addresses for my buffers in user space. 3. I give that address to the FPGA for DMA use. 4. When I get the FPGA interrupt, I invalidate the data cache and write the data to disk Does that sound like it would work? Would the address I receive from mmap() and pass to the FPGA be the actual physical address, or would I need to send the physical address to the FPGA and use the mmap() address to access and write to disk? One more question about this approach: does the mmap() call prevent the kernel from using this memory for other purposes? Will the kernel be able to move this memory elsewhere? I guess what I am asking is if this memory is locked for all other purposes? ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
RE: DMA to User-Space
Jonathan, what kind of memory is fpga_drv.strmdata? memory from the
kernel stack, something you allocated? You'll notice you were trying to
access location 0.
To use dma streaming memory you must call dma_map_single() to get the
physical address of fpga_drv.strmdata. That memory must be contiguous.
Then you may call remap_pfn_range with the page offset of the physical
address returned from the dma_map_single() call. Then when the cpu
accesses that region you must sync the region for with dma direction for
cpu or when the device needs to access it, for the device.
LDD3 seems a bit dated on this topic it says to use nopage instead of
remap_pfn_range(), in my experiments remap_pfn_range seemed ok. I would
ask others on the list for their experience on the topic.
Ultimately I went with mapping user space provided address and
implementing dma with scatter-gather.
-Original Message-
From: linuxppc-dev-bounces+john.p.price=l-3com@lists.ozlabs.org
[mailto:linuxppc-dev-bounces+john.p.price=l-3com@lists.ozlabs.org]
On Behalf Of Jonathan Haws
Sent: Wednesday, November 04, 2009 12:40 PM
To: bo@windriver.com
Cc: linuxppc-dev@lists.ozlabs.org
Subject: RE: DMA to User-Space
Jonathan Haws wrote:
All,
I have what may be an unconventional question:
Our application consists of data being captured by an FPGA,
processed, and transferred to SDRAM. I simply give the FPGA an
address of where I want it stored in SDRAM and it simply DMAs the
data over and interrupts me when finished. I then take that data
and store it to disk.
I have code in user space that handles all of the writing to disk
nicely and fast enough for my application (I am capturing data at
about 35-40 Mbytes/sec).
My question is this: is it possible to give a user-space pointer
to the FPGA to DMA to? It seems like I would have problems with
alignment, address manipulation, and a whole slew of other issues.
What would be the best way to accomplish something like that? I
want to handle all the disk access in user-space, but I do not want
to have to copy 40 MB/s from kernel space to user-space either.
You can maintain a DMA buffer in kernel, then mmap to user space.
And
maybe you need some handshake between FPGA and the apps to balance
input
datas with datas to disk.
I can maintain an allocated, DMA-safe buffer in kernel space if
needed. Can I simply get a user-space pointer to that buffer? What
calls are needed to translate addresses?
Use remap_pfn_range() in your kernel DMA buffer manipulation driver
.mmap() handler to export DMA buffer address to user space.
Can you provide an example for how to do that? I have an mmap routine
to map BARs that the FPGA maintains and I can access those, however when
I try to map the DMA buffer and access what is in it, the system
crashes. Here is the mmap function I have right now:
/* fpga_mmap()
*
* Description:
* The purpose of this function is to serve as a 'file_operation'
* which maps different PCI resources into the calling processes
* memory space.
*
* NOTE: The file offset are in page size; i.e.:
* offset 0 in process's mmap syscall - BAR0
* offset 4096 in process's mmap syscall - BAR1
* offset 8192 in process's mmap syscall - BAR2
* offset 12288 - Streaming DMA buffer
*
* Arguments:
* struct file *filp -- struct representing an open
file
* struct vm_area_struct *vma -- struct representing memory
'segment'
*
* Returns:
* int -- indication of success or failure
*
*/
int fpga_mmap(struct file *filp, struct vm_area_struct *vma)
{
struct pci_dev *dev;
unsigned long addressToMap;
uint8_t mapType = 0; /* 0 = IO, 1 = memory */
/* Get the PCI device */
dev = (struct pci_dev*)(filp-private_data);
/* Map in the appropriate BAR based on page offset */
if (vma-vm_pgoff == FPGA_CONFIG_SPACE)
{
/* Map BAR1 (the CONFIG area) */
printk(KERN_ALERT FPGA: Mapping BAR1 (CONFIG BAR).\n);
addressToMap = pci_resource_start(dev,
FPGA_CONFIG_SPACE);
printk(KERN_ALERT FPGA: PCI BAR1 (CONFIG BAR) Size -
%#08x.\n,
pci_resource_len(dev, FPGA_CONFIG_SPACE));
mapType = 0;
}
else if(vma-vm_pgoff == FPGA_TEST_SPACE)
{
/* Map BAR2 (the TEST area) */
printk(KERN_ALERT FPGA: Mapping BAR2 (TEST BAR).\n);
addressToMap = (pci_resource_start(dev, FPGA_TEST_SPACE)
+
pci_resource_len(dev, FPGA_TEST_SPACE)) -
FPGA_TEST_LENGTH;
printk(KERN_ALERT FPGA: PCI BAR2 (TEST BAR) Size -
%#08x.\n,
pci_resource_len(dev, FPGA_TEST_SPACE));
mapType = 0;
}
else if(vma-vm_pgoff == 3)
{
addressToMap = (unsigned long
Re: DMA to User-Space
Jonathan Haws wrote: All, I have what may be an unconventional question: Our application consists of data being captured by an FPGA, processed, and transferred to SDRAM. I simply give the FPGA an address of where I want it stored in SDRAM and it simply DMAs the data over and interrupts me when finished. I then take that data and store it to disk. I have code in user space that handles all of the writing to disk nicely and fast enough for my application (I am capturing data at about 35-40 Mbytes/sec). My question is this: is it possible to give a user-space pointer to the FPGA to DMA to? It seems like I would have problems with alignment, address manipulation, and a whole slew of other issues. What would be the best way to accomplish something like that? I want to handle all the disk access in user-space, but I do not want to have to copy 40 MB/s from kernel space to user-space either. You can maintain a DMA buffer in kernel, then mmap to user space. And maybe you need some handshake between FPGA and the apps to balance input datas with datas to disk. I can maintain an allocated, DMA-safe buffer in kernel space if needed. Can I simply get a user-space pointer to that buffer? What calls are needed to translate addresses? Use remap_pfn_range() in your kernel DMA buffer manipulation driver .mmap() handler to export DMA buffer address to user space. Tony Thanks for the help! I am still a newbie when it comes to kernel programming, so I really appreciate the help! Jonathan ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev -- Tony Liu | Liu Bo - WIND RIVER | China Development Center Tel: 86-10-8477-8542 ext: 8542 | Fax: 86-10-64790367 (M): 86-136-7117-3612 Address: 15/F, Wangjing TowerB, Chaoyang District, Beijing, P.R.China ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
