Can a real-time userspace thread that has been attached to EVL make standard ioctl() calls on a Linux driver? If so, what would any disadvantage be?
-----Original Message----- From: Jan Kiszka <jan.kis...@siemens.com> Sent: Wednesday, February 9, 2022 11:29 PM To: Russell Johnson <russell.john...@kratosdefense.com>; xenomai@xenomai.org Subject: [External] - Re: DMA with EVL-attached Thread CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. On 10.02.22 00:00, Russell Johnson via Xenomai wrote: > Hello, > > I currently have a DMA driver written based on the Linux DMA API found here: > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.kernel.org_doc_Documentation_DMA-2DAPI-2DHOWTO.txt&d=DwICaQ&c=zeCCs5WLaN-HWPHrpXwbFoOqeS0G3NH2_2IQ_bzV13g&r=-WrlDvHLmZCY7pl-EOIRaJl9uZvKbG-G8KkkUcIDh2F7Jr6SYxS_pdf-orjkExnI&m=TD3loAB1nnZmlmTxjzzJAPpuX-VwabaCxkXF_17saY26S7iMmpWBDYk06CHq7hNt&s=O2RICPwTryktJfcMEPRxx4SYyj79T6zCPcht6VUAikc&e= > . I have multiple realtime threads in my userspace app that have to make > constant DMA transfers. I have not seen any issue with DMA throughput using > the Linux API. The question is, do I need to re-architect the dma driver to > add in the oob_ioctl() FD so that a userspace EVL attached thread can use it? > If the answer is yes, I have experimented a bit with doing that and ran into > an issue. I have created a userspace test app with a thread attached to EVL > and attempted to allocate, write, read, and free a dma buffer from the driver > I modified with the oob_ioctl() FD. Alloc, write, and read all appear to work > fine. However, calling free (which boils down to the kernel call > "dma_free_coherent" is throwing a warning in the console: > > [75787.696762] ------------[ cut here ]------------ [75787.696763] > WARNING: CPU: 9 PID: 25074 at kernel/dma/mapping.c:532 > dma_free_attrs+0x40/0x90 [75787.696772] Modules linked in: dma_buf_mgr(OE) > xfs libcrc32c sr_mod sd_mod cdrom t10_pi ast drm_vram_helper drm_kms_helper > syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm ixgbe > ahci libahci igb libata mdio crc32c_intel ptp pps_core i2c_algo_bit dca [last > unloaded: dma_buf_mgr] > [75787.696793] CPU: 9 PID: 25074 Comm: test_thread:250 Tainted: G W > OE 5.15.0evl+ #6 > [75787.696796] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, > BIOS 1.3 02/13/2018 [75787.696797] IRQ stage: EVL [75787.696799] RIP: > 0010:dma_free_attrs+0x40/0x90 [75787.696802] Code: 89 fc 53 48 83 ec > 08 48 8b 9f 38 02 00 00 4c 89 45 d0 48 85 db 48 0f 44 1d 4d ce 8b 01 > e8 88 23 80 00 85 c0 4c 8b 45 d0 74 02 <0f> 0b 4d 85 ed 74 16 48 85 db > 75 20 4c 89 f9 4c 89 ea 4c 89 f6 4c [75787.696804] RSP: > 0018:ffffb3fd40747d48 EFLAGS: 00010202 [75787.696807] RAX: > 0000000000000001 RBX: 0000000000000000 RCX: 00000000798fa000 > [75787.696809] RDX: ffff8dda798fa000 RSI: 0000000000000400 RDI: > ffff8ddb01b2f0d0 [75787.696810] RBP: ffffb3fd40747d78 R08: > 0000000000000000 R09: 00080000000000ff [75787.696811] R10: > 00000000807eb2a2 R11: 0000000000000033 R12: ffff8ddb01b2f0d0 > [75787.696813] R13: ffff8dda798fa000 R14: 0000000000000400 R15: > 00000000798fa000 [75787.696814] FS: 00007f355a0d7700(0000) > GS:ffff8dde6d880000(0000) knlGS:0000000000000000 [75787.696816] CS: > 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [75787.696818] CR2: > 0000000000000000 CR3: 00000001b5448001 CR4: 00000000003706e0 [75787.696819] > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [75787.696820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 [75787.696822] Call Trace: > [75787.696826] free_dma_buf+0x5d/0xb0 [dma_buf_mgr] [75787.696831] > dma_buf_mgr_oob_ioctl+0x159/0x396 [dma_buf_mgr] [75787.696835] > EVL_ioctl+0x46/0xa0 [75787.696841] > handle_pipelined_syscall+0x181/0x2d0 > [75787.696846] __pipeline_syscall+0xbc/0x230 [75787.696851] ? > sched_clock+0x9/0x10 [75787.696856] pipeline_syscall+0x34/0x100 > [75787.696860] syscall_enter_from_user_mode+0x39/0xf0 > [75787.696865] do_syscall_64+0x15/0xa0 [75787.696867] > entry_SYSCALL_64_after_hwframe+0x44/0xae > [75787.696872] RIP: 0033:0x7f355b0f2f9b [75787.696874] Code: Unable to > access opcode bytes at RIP 0x7f355b0f2f71. > [75787.696875] RSP: 002b:00007f355a0d6860 EFLAGS: 00000246 ORIG_RAX: > 000000000000009d [75787.696877] RAX: ffffffffffffffda RBX: > 00007f355a0d6900 RCX: 00007f355b0f2f9b [75787.696878] RDX: > 0000000040042501 RSI: 0000000000000006 RDI: 0000000010000002 > [75787.696880] RBP: 0000000040042501 R08: 0000000000000000 R09: > 0000000000020750 [75787.696881] R10: 00007f355a0d6900 R11: > 0000000000000246 R12: 0000000000000006 [75787.696882] R13: > 0000000000801000 R14: 0000000000000000 R15: 00007f355a0d7700 > [75787.696884] ---[ end trace 9678e85a266ff468 ]--- > > This warning does not exist when running the same functionality from the > normal ioctl() FD from a standard thread in Linux. Is what I am trying to do > possible? If so, any clue on what I am seeing here? Yes, classic mistake: You are calling a Linux, in-band-only function (free_dma_buf) from an RT, out-of-band context. That does not work. Linux resource management has to happen in in-band context (standard ioctl handler). Jan -- Siemens AG, Technology Competence Center Embedded Linux
