Async DMA operations with Xenomai 3.1

[Sylvan Williams via Xenomai]

Hi everyone,

I have some questions about which approach I should take to allow OOB execution 
of DMA completion handlers when running kernel 4.14.110 + IPipe + Xenomai 3.1.

The system is a Zynq (dual ARM Cortex A-9), and I have a UDD driver which 
registers a hardware GPIO interrupt that’s incoming from the FPGA. The UDD 
driver also performs mmap’ing of some memory regions that are mapped to DMA, 
and those regions are submitted to the dmaengine for transfer into FPGA block 
RAM inside the interrupt handler of the UDD driver. The DMA controller also has 
an interrupt which is tied into the GIC to signal the completion of an operation

This works, for a while, but eventually I end up with a system hang that I’m 
(guessing) is due to deadlock between the head and root stages for calling the 
completion handler inside the xilinx-dma driver.

I stumbled upon the EVL project and Dovetail just last week, and it appears 
Dovetail addresses this issue with the introduction of the IRQF_OOB flag. I’m 
in the process of attempting to add the same functionality to IPipe, but now 
I’m looking for advice for the best approach since I feel like I’m about to get 
myself into a pretty intense code tangle.

So my question is:
Should I continue with this approach? or
Scrap it, and create an RTDM version of the xilinx DMA driver?
Is there an easier approach I haven’t thought of?

For more context, below is a snippet of the DMA transaction which is called 
from the udd_device.handler. Again, I believe the problem lies in the fact 
dmaengine_submit() and dma_async_issue_pending are called from OOB context, 
while the completion handler inside the xilinx DMA driver needs to run on 
receipt of the hardware interrupt from the DMA controller.

Also, this is a closed system, so the solution doesn’t necessarily have/need to 
fit into the kernel in an upstreamable fashion, if that makes sense.

There’s also the possibility that I’m completely incorrect in my theory, so no 
harm in pointing that out if it’s the case.

Any advice will be much appreciated!

Thanks,

Sylvan


static int dma_tx(fermi_tx_buf_t buf)
{
    int err;
    struct dma_async_tx_descriptor* txd;
    dma_addr_t dma_handle_src;
    dma_addr_t dma_handle_dest;
    size_t transaction_len;
    int active_bank;

    /* The MSB of the BRAM address bus is connected to the EMIO interface of 
the processor.
       Reading the value will indicate which half of the BRAM buffer is 
currently being written
       to the CV DAC array. Transfers need to occur into the _inactive_ buffer. 
Read the value
       add set the destination address accordingly
    */
    active_bank = gpio_get_value(zynq_emio_active_bank);

    /* MSB high - transfer to low buffer */
    if (active_bank == true)
    {
        dma_handle_dest = (dma_addr_t)soundengine_params_dest_addr;
    }
    /* MSB low - transfer to high buffer */
    else
    {
        dma_handle_dest = (dma_addr_t)(soundengine_params_dest_addr + 
soundengine_buf_len);
    }

    /* Set DDR RAM source address and transaction length depending on the value 
passed through the ioctl */
    if (buf == BUF_A)
    {
        transaction_len = fermi_platform_dev->tx.transaction_len;
        dma_handle_src = 
dma_map_single(fermi_platform_dev->tx.chan->device->dev, 
fermi_platform_dev->tx.dma_buf_src, transaction_len, DMA_TO_DEVICE);
    }
    else if (buf == BUF_B)
    {
        transaction_len = fermi_platform_dev->tx.transaction_len;
        dma_handle_src = 
dma_map_single(fermi_platform_dev->tx.chan->device->dev,
                                                         
(void*)((uint8_t*)fermi_platform_dev->tx.dma_buf_src + soundengine_buf_len), 
transaction_len, DMA_TO_DEVICE);
    }
    else
    {
        dev_err(&fermi_platform_dev->pdev->dev, "Cyclic TX DMA - unknown 
argument!\n");
        goto fail;
    }

    fermi_platform_dev->tx.transaction_addr = dma_handle_src;
    fermi_platform_dev->tx.transaction_len = transaction_len;
    reinit_completion(&fermi_platform_dev->tx.cmd_complete);

    txd = dmaengine_prep_dma_memcpy(fermi_platform_dev->tx.chan, 
dma_handle_dest, dma_handle_src, transaction_len, DMA_PREP_INTERRUPT | 
DMA_CTRL_ACK);

    if (!txd)
    {
        dev_err(&fermi_platform_dev->pdev->dev, "Unable to prepare cyclic TX 
DMA transaction!\n");
        //dmaengine_terminate_all(fermi_platform_dev->tx.chan);
        err = -ENOMEM;
        goto fail;
    }

    txd->callback = tx_dma_callback;
    txd->callback_param = &fermi_platform_dev->tx;

    dmaengine_submit(txd);
    dma_async_issue_pending(fermi_platform_dev->tx.chan);

    return 0;

fail:
    return err;
}