RE: returned dma address value of dma_alloc_coherent, is it always in the lower 4GB range?
Hello Valdis,
>-Original Message-
>From: Valdis Kletnieks On Behalf Of Valdis Kl?tnieks
>Sent: Tuesday, March 14, 2023 2:15 PM
>To: Chan Kim
>Cc: kernelnewbies@kernelnewbies.org
>Subject: Re: returned dma address value of dma_alloc_coherent, is it always
>in the lower 4GB range?
>
>On Mon, 13 Mar 2023 21:28:28 +0900, "Chan Kim" said:
>
>> which is 32-bit addressable. Even if the device indicates (via the
>> DMA
>> mask)
>> that it may address the upper 32-bits, consistent allocation will only
>> return > 32-bit addresses for DMA if the consistent DMA mask has been
>> explicitly changed via dma_set_coherent_mask(). This is true of the
>> dma_pool interface as well.
>> --
>
>> By context I guess it returns the address in lower 32bit address range
>> even it the coherent_dma_mask is set to some bigger value.
>
>No, you have that backwards. It's not "even if". It wil give you a
32-bit
>address *UNLESS* the driver has called dma_set_coherent_mask() to set a
>wider mask *and* the device indicates it supports more than 32 bit
>addresses...
I'm sorry but please allow me to ask one more time.
1. "The consistent DMA mapping interfaces, will by default return a DMA
address
which is 32-bit addressable."
==> so by default consistent dma mapping API returns 32-bit address, and
this 32-bit address means dma address, not physical address. Am I correct?
(because the dma address will be translated to physical address by the iommu
anyway)
2. " Even if the device indicates (via the DMA mask)
that it may address the upper 32-bits, consistent allocation will only
return > 32-bit addresses for DMA if the consistent DMA mask has been
explicitly changed via dma_set_coherent_mask(). This is true of the
dma_pool interface as well."
==> So I now understand the first 'even if' is for streaming dma mask
('dma_mask' in the struct device), and the second 'if' is for the
'coherent_dma_mask' in the struct device. And when coherent_dma_mask is not
set, by default, the coherent mapping returns 32 bit address, but if the
coherent_dma_mask is set to use wider than 32 bit addresses, it returns the
wider address which seems obvious now.
Please correct me if I'm wrong.
Thank you!
Chan Kim
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: returned dma address value of dma_alloc_coherent, is it always in the lower 4GB range?
On Mon, 13 Mar 2023 21:28:28 +0900, "Chan Kim" said: > which is 32-bit addressable. Even if the device indicates (via the DMA > mask) > that it may address the upper 32-bits, consistent allocation will only > return > 32-bit addresses for DMA if the consistent DMA mask has been > explicitly changed via dma_set_coherent_mask(). This is true of the > dma_pool interface as well. > -- > By context I guess it returns the address in lower 32bit address range even > it the coherent_dma_mask is set to some bigger value. No, you have that backwards. It's not "even if". It wil give you a 32-bit address *UNLESS* the driver has called dma_set_coherent_mask() to set a wider mask *and* the device indicates it supports more than 32 bit addresses... pgptUYVQP566D.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
returned dma address value of dma_alloc_coherent, is it always in the lower 4GB range?
Hello experts, I was reading the well known DMA-API-HOWTO document (https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt), (I've read this document a couple of times before in the past.) In the " Using Consistent DMA mappings" section it says : - dma_addr_t dma_handle; cpu_addr = dma_alloc_coherent(dev, size, _handle, gfp); ... The consistent DMA mapping interfaces, will by default return a DMA address which is 32-bit addressable. Even if the device indicates (via the DMA mask) that it may address the upper 32-bits, consistent allocation will only return > 32-bit addresses for DMA if the consistent DMA mask has been explicitly changed via dma_set_coherent_mask(). This is true of the dma_pool interface as well. -- I understand by 'DMA address', it means dma_handle (the address the device will use for DMA). I can't understand the expression "consistent allocation will only return > 32-bit addresses". Does it mean that when the driver sets the coherent_dma_mask, for example, to DMA_BIT_MAST(64) and it was successful, does it return the dma_handle in lower 32 bit address? What does "return > 32-bit addresses" mean? The ">" syntax here is so obscure with no clear definition of usage. By context I guess it returns the address in lower 32bit address range even it the coherent_dma_mask is set to some bigger value. (And I know generally, dma address(= bus address) is not equal to the physical address and is assigned by the OS for convenience) Thank you. Chan Kim ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
RE: does dma_alloc_coherent setup iommu page table?
I realized the IOMMU mapping is done with dma_alloc_ or dma_map_.. It's so basic (I have read https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt ) but I forgot. Thanks! Chan Kim From: c...@etri.re.kr Sent: Wednesday, September 29, 2021 4:15 PM To: kernelnewbies@kernelnewbies.org Subject: does dma_alloc_coherent setup iommu page table? Hello all, When the driver calls dma_alloc_coherent for a device, and if the device uses iommu, is the table entry for the iommu setup for the dma_handle? Seeing the struct device has something related to iommu, I guess so.. Any help appreciated. Thanks! Chan Kim ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
does dma_alloc_coherent setup iommu page table?
Hello all, When the driver calls dma_alloc_coherent for a device, and if the device uses iommu, is the table entry for the iommu setup for the dma_handle? Seeing the struct device has something related to iommu, I guess so.. Any help appreciated. Thanks! Chan Kim ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Difference dma_alloc_coherent() in x86_32 and x86_64
Hello!
I have a driver for PCI device which uses CMA framework for allocating
big coherent blocks of memory for DMA.
Allocation looks like:
typedef struct {
struct list_head list;
uint32_t size8;
void *kaddr;
dma_addr_t paddr;
} dma_region_t;
LIST_HEAD(region_list);
..
void* AllocDMA( size_t size )
{
dma_region_t *new_region;
new_region = kmalloc(sizeof(dma_region_t), GFP_KERNEL);
new_region->size8 = size;
new_region->kaddr = dma_alloc_coherent( NULL, size,
_region->paddr, GFP_KERNEL | GFP_DMA32 );
list_add(_region->list, _list);
printk("pcidev: cma_alloc paddr %pad kaddr %p size %d\n",
_region->paddr, new_region->kaddr, new_region->size8);
return new_region->kaddr;
}
It works fine on kernel 3.18.26 in 32bit mode.
When I reconfigure same kernel to run in 64bit mode(enabling in
'menuconfig' option '64-bit kernel') and trying to use this driver with
it
allocation failed with message in dmesg:
...
[ 1393.835535] fallback device: swiotlb buffer is full (sz: 8388608
bytes)
[ 1393.835579] pcidev: cma_alloc paddr 0x880234861220 kaddr
(null) size 8388608
...
My kernel command line is: swiotlb=16384 iommu=soft cma=256M
Anyone can explain this strange behavior?
--
D
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: dma_alloc_coherent
Hello Saumendra, I would suggest you to go through the following link: 1) This will explain need of DMA with practical scenario 2) ARM DMA Mapping with example https://sanjeevsharmaengg.wordpress.com/2014/08/26/why-dma-required-in-linux-kernel/ http://linuxkernelhacker.blogspot.in/2014/07/arm-dma-mapping-explained.html I hope these link will give you more confidence around understanding of DMA. Regards Sanjeev Sharma On Mon, Feb 8, 2016 at 12:29 PM, Saumendra Dash <saumendr...@hcl.com> wrote: > >> Hi Ran, > > the api which you have mentioned... > > > >> void * > >> dma_alloc_coherent(struct device *dev, size_t size, > >> dma_addr_t *dma_handle, gfp_t flag) > >> > >> is the kernel api to alloc consistent memory. > >> > >> DMA devices understand the physical addresses not the virtual addresses. > >> > >> which means that we must supply to the dma device, the physical > >> address, to read > >> > >> from or to write to. > >> > >> The second argument of this api is an input argument which is updated > >> by the kernel if this api returns a success (and contains the physical > >> base address of the allocated memory) and the returned value of this > >> api is the kernel virtual address. > >> > >> if the *CPU* has to operate on this memory (assume that the memory is > >> dma'ed by the dma device and cpu want to read it for further > >> processing ) it should use the virtual address, so the returned value > >> of this api, as the base address. > >> However, if the dma device has to operate on this memory (assume > >> device want to write to this memory), it should use the *dma_handle* , > >> which is the physical address (base) of the dma memory. > >> > >> Now the question is how the dma device knows about this *physical* > address? > >> The answer is that the "dma controller" register would have a register > >> to accept this physical address. > >> > >> So the sequence of steps probably would be, in your case: > >> 1: allocate the dma memory > >> 2: programme the dma controller register with the physical address > >> returned by this api, plus the size of the transaction and may be some > >> more registers for setting some kind of flags (depends on your dma > >> device) > >> 3: programme the dma controller's dma *start* bit. > >> > >> after this the dma starts and dma device starts writing to the memory . > > Hi Vishwas, > > >That's fully clarify the questions about dma_alloc_coherent. > > >I also try to figure out what's the difference between dma_alloc_coherent > and dma_map_single. > > >I could not find and important difference between these two methods. > >1. With both methods I stil need to program the dma controller with the > physical address and the start trigger. > >2. I can still do the allocation whenever I want with both methods (for > example at the initialization of the driver), 3. Not sure what the actuall > dma_map_single does (and if it really necessary to use it), becuase it > >seems I could just translate the virtual value from kmalloc into physical > address and return it to the dma controller. > > DMA transfers are done either in BURST mode or CYCLE STEALING mode. > In Burst mode, the bus is captured for the entire duration of the > transfer from the SRC to DST. In this case, the bus will be released > when the Xfer is complete, so obviously it is not an efficient way of doing > DMA. DMA_ALLOC_COHERENT() does this way. > In Cycle Stealing mode, the DMA controller grab the bus when free, send a > byte and then free the bus immediately. This process is repeated till the > Xfer is complete, it is very efficient sine the bus is not grabbed for the > entire transaction to complete. DMA_MAP_SINGLE() does this way. > > Hope this helps. > > Thanks, > Saumendra > > > ::DISCLAIMER:: > > > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > E-mail transmission is not guaranteed to be secure or error-free as > information could be intercepted, corrupted, > lost, destroyed, arrive late or incomplete, or may contain viruses in > transmission. The e mail and its contents > (with or without referred errors) shall therefore not attach any liability > on the originator or HCL or its affiliates. > Views or opinions, if any, pre
Re: dma_alloc_coherent
Hi Ran, the api which you have mentioned... void * dma_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag) is the kernel api to alloc consistent memory. DMA devices understand the physical addresses not the virtual addresses. which means that we must supply to the dma device, the physical address, to read from or to write to. The second argument of this api is an input argument which is updated by the kernel if this api returns a success (and contains the physical base address of the allocated memory) and the returned value of this api is the kernel virtual address. if the *CPU* has to operate on this memory (assume that the memory is dma'ed by the dma device and cpu want to read it for further processing ) it should use the virtual address, so the returned value of this api, as the base address. However, if the dma device has to operate on this memory (assume device want to write to this memory), it should use the *dma_handle* , which is the physical address (base) of the dma memory. Now the question is how the dma device knows about this *physical* address? The answer is that the "dma controller" register would have a register to accept this physical address. So the sequence of steps probably would be, in your case: 1: allocate the dma memory 2: programme the dma controller register with the physical address returned by this api, plus the size of the transaction and may be some more registers for setting some kind of flags (depends on your dma device) 3: programme the dma controller's dma *start* bit. after this the dma starts and dma device starts writing to the memory . I hope, this clarifies you. On Sun, Feb 7, 2016 at 10:30 PM, <kernelnewbies-requ...@kernelnewbies.org> wrote: > Send Kernelnewbies mailing list submissions to > kernelnewbies@kernelnewbies.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies > or, via email, send a message with subject or body 'help' to > kernelnewbies-requ...@kernelnewbies.org > > > You can reach the person managing the list at > kernelnewbies-ow...@kernelnewbies.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Kernelnewbies digest..." > > > Today's Topics: > >1. Re: dma_alloc_coherent (Ran Shalit) > > > -- > > Message: 1 > Date: Sat, 6 Feb 2016 21:22:49 +0200 > From: Ran Shalit <ransha...@gmail.com> > Subject: Re: dma_alloc_coherent > To: Denis Kirjanov <kirja...@gmail.com> > Cc: kernelnewbies <kernelnewbies@kernelnewbies.org> > Message-ID: > < > caj2omhli9lv6qq-yr1ynkgobwc_et-qtwdaggrdc9bgzlkn...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Feb 5, 2016 at 11:15 AM, Denis Kirjanov <kirja...@gmail.com> > wrote: > > On 2/5/16, Ran Shalit <ransha...@gmail.com> wrote: > >> Hello, > >> > >> I read the readme about dma API, but still don't understand how it > >> should be used > >> It is said that dma_alloc_coherent is responsible for allocating the > >> buffer. > > > >> > >> 1. But how to trigger the dma transaction to start ? > >> 2. Is there a callback when it is finished ? > > > > It's used for data transfer between IO device and system memory. You > > allocate a kernel buffer for DMA transaction (in this case dma from > > device to system memory) and setup your device for dma transfer. An IO > > device usually generates an interrupt when DMA transfer completes. > > Denis > > If I understand correctly the full picture how to use dma is as > following (schematics): > > buf = dma_alloc_coherent(); <<-- done once at the lifetime > > every time we need to trigger dma: > > //start transaction now > writeb(buf, DMA_ADDR) <<- juat an example, actually it is totally > depends on dma device > writeb(START_DMA, DMA_ADDR+2) <<- juat an example, actually it is > totally depends on dma device > > //usually we also register on irq for callback on finishing the > transaction. > > hope I got it all correct, if you have any comments please add. > > Thanks, > Ran > > > > -- > > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies > > > End of Kernelnewbies Digest, Vol 63, Issue 7 > > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
RE: dma_alloc_coherent
>> Hi Ran, > the api which you have mentioned... > >> void * >> dma_alloc_coherent(struct device *dev, size_t size, >> dma_addr_t *dma_handle, gfp_t flag) >> >> is the kernel api to alloc consistent memory. >> >> DMA devices understand the physical addresses not the virtual addresses. >> >> which means that we must supply to the dma device, the physical >> address, to read >> >> from or to write to. >> >> The second argument of this api is an input argument which is updated >> by the kernel if this api returns a success (and contains the physical >> base address of the allocated memory) and the returned value of this >> api is the kernel virtual address. >> >> if the *CPU* has to operate on this memory (assume that the memory is >> dma'ed by the dma device and cpu want to read it for further >> processing ) it should use the virtual address, so the returned value >> of this api, as the base address. >> However, if the dma device has to operate on this memory (assume >> device want to write to this memory), it should use the *dma_handle* , >> which is the physical address (base) of the dma memory. >> >> Now the question is how the dma device knows about this *physical* address? >> The answer is that the "dma controller" register would have a register >> to accept this physical address. >> >> So the sequence of steps probably would be, in your case: >> 1: allocate the dma memory >> 2: programme the dma controller register with the physical address >> returned by this api, plus the size of the transaction and may be some >> more registers for setting some kind of flags (depends on your dma >> device) >> 3: programme the dma controller's dma *start* bit. >> >> after this the dma starts and dma device starts writing to the memory . Hi Vishwas, >That's fully clarify the questions about dma_alloc_coherent. >I also try to figure out what's the difference between dma_alloc_coherent and >dma_map_single. >I could not find and important difference between these two methods. >1. With both methods I stil need to program the dma controller with the >physical address and the start trigger. >2. I can still do the allocation whenever I want with both methods (for >example at the initialization of the driver), 3. Not sure what the actuall >dma_map_single does (and if it really necessary to use it), becuase it >seems >I could just translate the virtual value from kmalloc into physical address >and return it to the dma controller. DMA transfers are done either in BURST mode or CYCLE STEALING mode. In Burst mode, the bus is captured for the entire duration of the transfer from the SRC to DST. In this case, the bus will be released when the Xfer is complete, so obviously it is not an efficient way of doing DMA. DMA_ALLOC_COHERENT() does this way. In Cycle Stealing mode, the DMA controller grab the bus when free, send a byte and then free the bus immediately. This process is repeated till the Xfer is complete, it is very efficient sine the bus is not grabbed for the entire transaction to complete. DMA_MAP_SINGLE() does this way. Hope this helps. Thanks, Saumendra ::DISCLAIMER:: The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: dma_alloc_coherent
On Fri, Feb 5, 2016 at 11:15 AM, Denis Kirjanov <kirja...@gmail.com> wrote: > On 2/5/16, Ran Shalit <ransha...@gmail.com> wrote: >> Hello, >> >> I read the readme about dma API, but still don't understand how it >> should be used >> It is said that dma_alloc_coherent is responsible for allocating the >> buffer. > >> >> 1. But how to trigger the dma transaction to start ? >> 2. Is there a callback when it is finished ? > > It's used for data transfer between IO device and system memory. You > allocate a kernel buffer for DMA transaction (in this case dma from > device to system memory) and setup your device for dma transfer. An IO > device usually generates an interrupt when DMA transfer completes. > Denis If I understand correctly the full picture how to use dma is as following (schematics): buf = dma_alloc_coherent(); <<-- done once at the lifetime every time we need to trigger dma: //start transaction now writeb(buf, DMA_ADDR) <<- juat an example, actually it is totally depends on dma device writeb(START_DMA, DMA_ADDR+2) <<- juat an example, actually it is totally depends on dma device //usually we also register on irq for callback on finishing the transaction. hope I got it all correct, if you have any comments please add. Thanks, Ran ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: dma_alloc_coherent
On 2/5/16, Ran Shalit <ransha...@gmail.com> wrote: > Hello, > > I read the readme about dma API, but still don't understand how it > should be used > It is said that dma_alloc_coherent is responsible for allocating the > buffer. > > 1. But how to trigger the dma transaction to start ? > 2. Is there a callback when it is finished ? It's used for data transfer between IO device and system memory. You allocate a kernel buffer for DMA transaction (in this case dma from device to system memory) and setup your device for dma transfer. An IO device usually generates an interrupt when DMA transfer completes. > > Thank you, > Ran > > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies > -- Regards / Mit besten Grüßen, Denis ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
dma_alloc_coherent
Hello, I read the readme about dma API, but still don't understand how it should be used It is said that dma_alloc_coherent is responsible for allocating the buffer. 1. But how to trigger the dma transaction to start ? 2. Is there a callback when it is finished ? Thank you, Ran ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
dma_alloc_coherent
In dma_alloc_coherent(), where is it allocating memory from, and how does it know that that memory is cache coherent? Does every device have it's cache coherent memory? I got lost at function pointer struct dma_map_ops ops, is there an easy way to figure out what ops-alloc_coherent() points to? ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: dma_alloc_coherent
On Wed, Oct 12, 2011 at 11:47 PM, bob jonny ilikepie...@live.com wrote:
In dma_alloc_coherent(), where is it allocating memory from, and how does
it know that that memory is cache coherent? Does every device have it's
cache coherent memory? I got lost at function pointer struct dma_map_ops
ops, is there an easy way to figure out what ops-alloc_coherent() points
to?
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Bob,
dma_alloc_coherent() invokes the alloc_coherent method for the particular
device which is the first parameter.
So to see from where the memory is getting allocated, you need to see that
device's dma_map_op's alloc_coherent method.
I went through some of them, they tend to alloc memory by calling either
__get_free_pages() or alloc_pages() functions.
To maintain cache coherency, the pages which are allocated in these
functions have to be uncached and if they where cached earlier then they
have to be flushed.
Following function explains all these : -
void *dma_generic_alloc_coherent(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp)
{
void *ret, *ret_nocache;
int order = get_order(size);
gfp |= __GFP_ZERO;
ret = (void *)__get_free_pages(gfp, order);
if (!ret)
return NULL;
/*
* Pages from the page allocator may have data present in
* cache. So flush the cache before using uncached memory.
*/
dma_cache_sync(dev, ret, size, DMA_BIDIRECTIONAL);
ret_nocache = (void __force *)ioremap_nocache(virt_to_phys(ret),
size);
if (!ret_nocache) {
free_pages((unsigned long)ret, order);
return NULL;
}
split_page(pfn_to_page(virt_to_phys(ret) PAGE_SHIFT), order);
*dma_handle = virt_to_phys(ret);
return ret_nocache;
}
Regards,
Rohan Puri
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
