PR #22566 opened by t-boiko URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22566 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22566.patch
When mapping Vulkan Video frames to DMA-BUF, synchronize using an exportable binary semaphore and sync_fd where supported, then attach the sync file to each exported buffer with DMA_BUF_IOCTL_IMPORT_SYNC_FILE; otherwise keep using WaitSemaphores. Track the export semaphore on a dedicated exec context so that dependencies are not dropped early. Export dma-buf fds with GetMemoryFdKHR for each populated f->mem[i], iterating up to the sw_format plane count instead of stopping at the image count, so multi-memory bindings are not skipped. Describe DRM layers using max(sw planes, image count) and query subresource layout with the correct aspect and image index when one VkImage backs multiple planes. Reference the source hw_frames_ctx on the mapped frame and close dma-buf fds on failure paths. For DMA-BUF-capable pools, honor VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT from format export queries when binding memory. With DRM modifiers and a video profile in create_pnext, preserve caller usage and image flags instead of overwriting them from generic supported_usage probing; use the modifier list create info when probing export flags for modifier tiling. In vulkan_decode, emit one image memory barrier per plane when the aspect mask contains multiple plane bits for output and reference images, and enlarge the barrier array for the worst case. For the decode output image view, OR in VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR from the output frames context's usage with DST (fixes VUID-VkVideoBeginCodingInfoKHR-slotIndex-07245) instead of adding DPB usage only when !is_current. In ff_vk_decode_add_slice, pass VkVideoProfileListInfoKHR (from the output frames context's create_pnext) as the pNext argument to ff_vk_get_pooled_buffer instead of the full create_pnext chain. In ff_vk_frame_params, set tiling to OPTIMAL only when it is not already DRM_FORMAT_MODIFIER_EXT. In ff_vk_decode_init, when the output pool's create_pnext includes VkImageDrmFormatModifierListCreateInfoEXT, initialize the DPB pool with that modifier-list pNext and DRM_FORMAT_MODIFIER_EXT tiling; otherwise use VkVideoProfileListInfoKHR and OPTIMAL as before. When VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR is unset, the output and DPB pools cannot use different layouts or tiling, so the DPB pool must match the output pool. Signed-off-by: Tymur Boiko <[email protected]> >From 31ef1696c8e2936a9dc209644f6606559a291dff Mon Sep 17 00:00:00 2001 From: Tymur Boiko <[email protected]> Date: Tue, 27 Jan 2026 18:16:33 +0100 Subject: [PATCH] vulkan: fix DRM map, decode barriers, and video frame setup for modifier output When mapping Vulkan Video frames to DMA-BUF, synchronize using an exportable binary semaphore and sync_fd where supported, then attach the sync file to each exported buffer with DMA_BUF_IOCTL_IMPORT_SYNC_FILE; otherwise keep using WaitSemaphores. Track the export semaphore on a dedicated exec context so that dependencies are not dropped early. Export dma-buf fds with GetMemoryFdKHR for each populated f->mem[i], iterating up to the sw_format plane count instead of stopping at the image count, so multi-memory bindings are not skipped. Describe DRM layers using max(sw planes, image count) and query subresource layout with the correct aspect and image index when one VkImage backs multiple planes. Reference the source hw_frames_ctx on the mapped frame and close dma-buf fds on failure paths. For DMA-BUF-capable pools, honor VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT from format export queries when binding memory. With DRM modifiers and a video profile in create_pnext, preserve caller usage and image flags instead of overwriting them from generic supported_usage probing; use the modifier list create info when probing export flags for modifier tiling. In vulkan_decode, emit one image memory barrier per plane when the aspect mask contains multiple plane bits for output and reference images, and enlarge the barrier array for the worst case. For the decode output image view, OR in VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR from the output frames context's usage with DST (fixes VUID-VkVideoBeginCodingInfoKHR-slotIndex-07245) instead of adding DPB usage only when !is_current. In ff_vk_decode_add_slice, pass VkVideoProfileListInfoKHR (from the output frames context's create_pnext) as the pNext argument to ff_vk_get_pooled_buffer instead of the full create_pnext chain. In ff_vk_frame_params, set tiling to OPTIMAL only when it is not already DRM_FORMAT_MODIFIER_EXT. In ff_vk_decode_init, when the output pool's create_pnext includes VkImageDrmFormatModifierListCreateInfoEXT, initialize the DPB pool with that modifier-list pNext and DRM_FORMAT_MODIFIER_EXT tiling; otherwise use VkVideoProfileListInfoKHR and OPTIMAL as before. When VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR is unset, the output and DPB pools cannot use different layouts or tiling, so the DPB pool must match the output pool. Signed-off-by: Tymur Boiko <[email protected]> --- libavcodec/vulkan_decode.c | 148 +++++++++++++++-------- libavutil/hwcontext_vulkan.c | 224 ++++++++++++++++++++++++++++++++--- 2 files changed, 305 insertions(+), 67 deletions(-) diff --git a/libavcodec/vulkan_decode.c b/libavcodec/vulkan_decode.c index 9ab8d45aa9..35349311bc 100644 --- a/libavcodec/vulkan_decode.c +++ b/libavcodec/vulkan_decode.c @@ -233,7 +233,8 @@ int ff_vk_decode_prepare_frame(FFVulkanDecodeContext *dec, AVFrame *pic, (AVVkFrame *)pic->data[0], hwfc->format[0], VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR | - (!is_current ? VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR : 0)); + (hwfc->usage & VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR)); + // the above fixes VUID-VkVideoBeginCodingInfoKHR-slotIndex-07245 if (err < 0) return err; @@ -335,12 +336,16 @@ int ff_vk_decode_add_slice(AVCodecContext *avctx, FFVulkanDecodePicture *vp, * easier, and gives us ample headroom. */ buf_size = 2 << av_log2(buf_size); + VkVideoProfileListInfoKHR *profile_list = (VkVideoProfileListInfoKHR *) + ff_vk_find_struct(ctx->s.hwfc->create_pnext, + VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR); + err = ff_vk_get_pooled_buffer(&ctx->s, &ctx->buf_pool, &new_ref, DECODER_IS_SDR(avctx->codec_id) ? (VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT) : VK_BUFFER_USAGE_VIDEO_DECODE_SRC_BIT_KHR, - ctx->s.hwfc->create_pnext, buf_size, + profile_list, buf_size, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | (DECODER_IS_SDR(avctx->codec_id) ? VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT : 0x0)); @@ -499,7 +504,8 @@ int ff_vk_decode_frame(AVCodecContext *avctx, .sType = VK_STRUCTURE_TYPE_VIDEO_END_CODING_INFO_KHR, }; - VkImageMemoryBarrier2 img_bar[37]; + /* Worst case: 3 output planes + maxActiveReferencePictures * 3 ref planes (e.g. H.264 16 refs) */ + VkImageMemoryBarrier2 img_bar[64]; int nb_img_bar = 0; size_t data_size = FFALIGN(vp->slices_size, ctx->caps.minBitstreamBufferSizeAlignment); @@ -565,29 +571,44 @@ int ff_vk_decode_frame(AVCodecContext *avctx, if (err < 0) return err; - /* Output image - change layout, as it comes from a pool */ - img_bar[nb_img_bar] = (VkImageMemoryBarrier2) { - .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, - .pNext = NULL, - .srcStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, - .dstStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, - .srcAccessMask = VK_ACCESS_2_NONE, - .dstAccessMask = VK_ACCESS_2_VIDEO_DECODE_WRITE_BIT_KHR, - .oldLayout = vkf->layout[0], - .newLayout = (layered_dpb || vp->dpb_frame) ? - VK_IMAGE_LAYOUT_VIDEO_DECODE_DST_KHR : - VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHR, /* Spec, 07252 utter madness */ - .srcQueueFamilyIndex = vkf->queue_family[0], - .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, - .image = vkf->img[0], - .subresourceRange = (VkImageSubresourceRange) { - .aspectMask = vp->view.aspect[0], - .layerCount = 1, - .levelCount = 1, - }, + /* Output image - change layout, as it comes from a pool. + * Use one barrier per plane when aspect has multiple plane bits so drivers + * that expect a single plane in GetPlaneAspect() (e.g. during layout + * transition/clear) do not assert. */ + + static const VkImageAspectFlags plane_bits[] = { + VK_IMAGE_ASPECT_PLANE_0_BIT, VK_IMAGE_ASPECT_PLANE_1_BIT, VK_IMAGE_ASPECT_PLANE_2_BIT }; - ff_vk_exec_update_frame(&ctx->s, exec, pic, - &img_bar[nb_img_bar], &nb_img_bar); + VkImageAspectFlags aspect = vp->view.aspect[0]; + int is_multiplane = aspect & (VK_IMAGE_ASPECT_PLANE_0_BIT | VK_IMAGE_ASPECT_PLANE_1_BIT | VK_IMAGE_ASPECT_PLANE_2_BIT); + + const int n = (is_multiplane ? FF_ARRAY_ELEMS(plane_bits) : 1); + for (int p = 0; p < n; p++) { + if (is_multiplane && !(aspect & plane_bits[p])) + continue; + img_bar[nb_img_bar] = (VkImageMemoryBarrier2) { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .pNext = NULL, + .srcStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, + .dstStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, + .srcAccessMask = VK_ACCESS_2_NONE, + .dstAccessMask = VK_ACCESS_2_VIDEO_DECODE_WRITE_BIT_KHR, + .oldLayout = vkf->layout[0], + .newLayout = (layered_dpb || vp->dpb_frame) ? + VK_IMAGE_LAYOUT_VIDEO_DECODE_DST_KHR : + VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHR, + .srcQueueFamilyIndex = vkf->queue_family[0], + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = vkf->img[0], + .subresourceRange = (VkImageSubresourceRange) { + .aspectMask = is_multiplane ? plane_bits[p] : aspect, + .layerCount = 1, + .levelCount = 1, + }, + }; + ff_vk_exec_update_frame(&ctx->s, exec, pic, + &img_bar[nb_img_bar], &nb_img_bar); + } /* Reference for the current image, if existing and not layered */ if (vp->dpb_frame) { @@ -622,28 +643,35 @@ int ff_vk_decode_frame(AVCodecContext *avctx, if (!rvp->dpb_frame) { AVVkFrame *rvkf = (AVVkFrame *)ref->data[0]; + VkImageAspectFlags aspect_ref = rvp->view.aspect_ref[0]; + int ref_multiplane = aspect_ref & (VK_IMAGE_ASPECT_PLANE_0_BIT | VK_IMAGE_ASPECT_PLANE_1_BIT | VK_IMAGE_ASPECT_PLANE_2_BIT); + const int n_ref = (ref_multiplane ? FF_ARRAY_ELEMS(plane_bits) : 1); - img_bar[nb_img_bar] = (VkImageMemoryBarrier2) { - .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, - .pNext = NULL, - .srcStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, - .dstStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, - .srcAccessMask = VK_ACCESS_2_NONE, - .dstAccessMask = VK_ACCESS_2_VIDEO_DECODE_READ_BIT_KHR | - VK_ACCESS_2_VIDEO_DECODE_WRITE_BIT_KHR, - .oldLayout = rvkf->layout[0], - .newLayout = VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHR, - .srcQueueFamilyIndex = rvkf->queue_family[0], - .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, - .image = rvkf->img[0], - .subresourceRange = (VkImageSubresourceRange) { - .aspectMask = rvp->view.aspect_ref[0], - .layerCount = 1, - .levelCount = 1, - }, - }; - ff_vk_exec_update_frame(&ctx->s, exec, ref, - &img_bar[nb_img_bar], &nb_img_bar); + for (int p = 0; p < n_ref; p++) { + if (ref_multiplane && !(aspect_ref & plane_bits[p])) + continue; + img_bar[nb_img_bar] = (VkImageMemoryBarrier2) { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .pNext = NULL, + .srcStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, + .dstStageMask = VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR, + .srcAccessMask = VK_ACCESS_2_NONE, + .dstAccessMask = VK_ACCESS_2_VIDEO_DECODE_READ_BIT_KHR | + VK_ACCESS_2_VIDEO_DECODE_WRITE_BIT_KHR, + .oldLayout = rvkf->layout[0], + .newLayout = VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHR, + .srcQueueFamilyIndex = rvkf->queue_family[0], + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = rvkf->img[0], + .subresourceRange = (VkImageSubresourceRange) { + .aspectMask = ref_multiplane ? plane_bits[p] : aspect_ref, + .layerCount = 1, + .levelCount = 1, + }, + }; + ff_vk_exec_update_frame(&ctx->s, exec, ref, + &img_bar[nb_img_bar], &nb_img_bar); + } } } } else if (vp->decode_info.referenceSlotCount || @@ -1217,7 +1245,13 @@ int ff_vk_frame_params(AVCodecContext *avctx, AVBufferRef *hw_frames_ctx) } } - hwfc->tiling = VK_IMAGE_TILING_OPTIMAL; + if (hwfc->tiling != VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT) { + hwfc->tiling = VK_IMAGE_TILING_OPTIMAL; + } + + av_log(avctx, AV_LOG_VERBOSE, "Vulkan frame params: tiling=%u (0=OPTIMAL 1=LINEAR 1000158000=DRM_MOD)\n", + (unsigned)hwfc->tiling); + hwfc->usage = VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_SAMPLED_BIT; @@ -1382,10 +1416,24 @@ int ff_vk_decode_init(AVCodecContext *avctx) dpb_frames->height = s->frames->height; dpb_hwfc = dpb_frames->hwctx; - dpb_hwfc->create_pnext = (void *)ff_vk_find_struct(ctx->s.hwfc->create_pnext, - VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR); + void *profile_list = (void *)ff_vk_find_struct(ctx->s.hwfc->create_pnext, + VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR); + /* Reference (DPB) images use the same tiling and pNext chain as output. + * If VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR is 0, the + * driver does not support separate output and DPB with different layouts/tiling. */ + void *drm_create_pnext = (void *)ff_vk_find_struct(ctx->s.hwfc->create_pnext, + VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT); + if (drm_create_pnext) { + dpb_hwfc->create_pnext = drm_create_pnext; + dpb_hwfc->tiling = VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT; + if (!ff_vk_find_struct(drm_create_pnext, VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR)) { + av_log(avctx, AV_LOG_WARNING, "DRM modifier list found in create_pnext, but no video profile list found\n"); + } + } else { + dpb_hwfc->create_pnext = profile_list; + dpb_hwfc->tiling = VK_IMAGE_TILING_OPTIMAL; + } dpb_hwfc->format[0] = s->hwfc->format[0]; - dpb_hwfc->tiling = VK_IMAGE_TILING_OPTIMAL; dpb_hwfc->usage = VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR; if (ctx->common.layered_dpb) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 65e2256e2d..4f93ea5131 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -178,6 +178,9 @@ typedef struct VulkanFramesPriv { /* Image conversions */ FFVkExecPool compute_exec; + /* DRM sync-fd export submit only; avoids sharing with upload_exec so dep is never discarded without wait */ + FFVkExecPool export_exec; + /* Image transfers */ FFVkExecPool upload_exec; FFVkExecPool download_exec; @@ -190,6 +193,9 @@ typedef struct VulkanFramesPriv { /* Properties for DRM modifier for each plane in the image */ VkDrmFormatModifierPropertiesEXT drm_format_modifier_properties[5]; + + /* Set when physical device reports DEDICATED_ONLY for DMA-BUF export (try_export_flags) */ + int export_requires_dedicated; } VulkanFramesPriv; typedef struct AVVkFrameInternal { @@ -2476,6 +2482,10 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, vk->GetImageMemoryRequirements2(hwctx->act_dev, &req_desc, &req); + av_log(hwfc, AV_LOG_VERBOSE, + "[VULKAN alloc] plane %d: driver reports prefersDedicatedAllocation=%d requiresDedicatedAllocation=%d\n", + img_cnt, (int)ded_req.prefersDedicatedAllocation, (int)ded_req.requiresDedicatedAllocation); + if (f->tiling == VK_IMAGE_TILING_LINEAR) req.memoryRequirements.size = FFALIGN(req.memoryRequirements.size, p->props.properties.limits.minMemoryMapAlignment); @@ -2483,6 +2493,8 @@ static int alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f, /* In case the implementation prefers/requires dedicated allocation */ use_ded_mem = ded_req.prefersDedicatedAllocation | ded_req.requiresDedicatedAllocation; + if (((VulkanFramesPriv *)hwfc->hwctx)->export_requires_dedicated) + use_ded_mem = 1; if (use_ded_mem) ded_alloc.image = f->img[img_cnt]; @@ -2854,7 +2866,8 @@ static void try_export_flags(AVHWFramesContext *hwfc, .type = VK_IMAGE_TYPE_2D, .tiling = hwctx->tiling, .usage = hwctx->usage, - .flags = VK_IMAGE_CREATE_ALIAS_BIT, + .flags = (hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT && has_mods) ? + (hwctx->img_flags) : (VkImageCreateFlags)(VK_IMAGE_CREATE_ALIAS_BIT), }; nb_mods = has_mods ? drm_mod_info->drmFormatModifierCount : 1; @@ -2865,9 +2878,17 @@ static void try_export_flags(AVHWFramesContext *hwfc, ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx->phys_dev, &pinfo, &props); + av_log(hwfc, AV_LOG_VERBOSE, "GetPhysicalDeviceImageFormatProperties2: mod[%d]=0x%llx -> %s\n", + i, has_mods ? (unsigned long long)phy_dev_mod_info.drmFormatModifier : 0ULL, + ret == VK_SUCCESS ? "OK" : "FAIL"); if (ret == VK_SUCCESS) { *iexp |= exp; *comp_handle_types |= eprops.externalMemoryProperties.compatibleHandleTypes; + if (exp == VK_EXTERNAL_MEMORY_HANDLE_TYPE_DMA_BUF_BIT_EXT) { + VulkanFramesPriv *fp = hwfc->hwctx; + fp->export_requires_dedicated = !!(eprops.externalMemoryProperties.externalMemoryFeatures & + VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT); + } } } } @@ -2895,7 +2916,8 @@ static AVBufferRef *vulkan_pool_alloc(void *opaque, size_t size) ? VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT : VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT); #else - if (p->vkctx.extensions & FF_VK_EXT_EXTERNAL_FD_MEMORY) + if (p->vkctx.extensions & FF_VK_EXT_EXTERNAL_FD_MEMORY && + hwctx->tiling != VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT) try_export_flags(hwfc, &eiinfo.handleTypes, &e, VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT); @@ -2914,8 +2936,10 @@ static AVBufferRef *vulkan_pool_alloc(void *opaque, size_t size) err = create_frame(hwfc, &f, hwctx->tiling, hwctx->usage, hwctx->img_flags, hwctx->nb_layers, eiinfo.handleTypes ? &eiinfo : hwctx->create_pnext); - if (err) + if (err) { + av_log(hwfc, AV_LOG_ERROR, "vulkan_pool_alloc failed: create_frame failed: %d\n", err); return NULL; + } err = alloc_bind_mem(hwfc, f, eminfo, sizeof(*eminfo)); if (err) @@ -2943,6 +2967,7 @@ static AVBufferRef *vulkan_pool_alloc(void *opaque, size_t size) return avbuf; fail: + av_log(hwfc, AV_LOG_ERROR, "vulkan_pool_alloc failed with error %d\n", err); vulkan_frame_free(hwfc, f); return NULL; } @@ -2969,6 +2994,7 @@ static void vulkan_frames_uninit(AVHWFramesContext *hwfc) } ff_vk_exec_pool_free(&p->vkctx, &fp->compute_exec); + ff_vk_exec_pool_free(&p->vkctx, &fp->export_exec); ff_vk_exec_pool_free(&p->vkctx, &fp->upload_exec); ff_vk_exec_pool_free(&p->vkctx, &fp->download_exec); @@ -3044,7 +3070,14 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) } /* Lone DPB images do not need additional flags. */ - if (!is_lone_dpb) { + /* With DRM modifier + video profile the caller has already chosen a valid + * usage/img_flags/chain; do not add usage or img_flags (supported_usage does + * not consider the actual modifier or video profile). */ + int drm_mod_with_video = (hwctx->tiling == VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT && + ff_vk_find_struct(hwctx->create_pnext, + VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR)); + + if (!is_lone_dpb && !drm_mod_with_video) { /* Image usage flags */ hwctx->usage |= supported_usage & (VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | @@ -3111,6 +3144,11 @@ static int vulkan_frames_init(AVHWFramesContext *hwfc) if (err) return err; + err = ff_vk_exec_pool_init(&p->vkctx, p->compute_qf, &fp->export_exec, + p->compute_qf->num, 0, 0, 0, NULL); + if (err) + return err; + err = ff_vk_exec_pool_init(&p->vkctx, p->transfer_qf, &fp->upload_exec, p->transfer_qf->num*2, 0, 0, 0, NULL); if (err) @@ -4128,6 +4166,36 @@ static VkImageAspectFlags plane_index_to_aspect(int plane) { return VK_IMAGE_ASPECT_MEMORY_PLANE_0_BIT_EXT; } +static void export_sem_free(void *opaque, uint8_t *data) +{ + FFVulkanContext *s = opaque; + VkSemaphore sem = *(VkSemaphore *)data; + FFVulkanFunctions *vk = &s->vkfn; + vk->DestroySemaphore(s->hwctx->act_dev, sem, s->hwctx->alloc); + av_free(data); +} + +static int add_export_sem_dep(FFVulkanContext *ctx, FFVkExecContext *exec, + VkSemaphore *export_sem) +{ + VkSemaphore *sem_ptr = av_malloc(sizeof(VkSemaphore)); + if (!sem_ptr) + return AVERROR(ENOMEM); + *sem_ptr = *export_sem; + AVBufferRef *avbuf = av_buffer_create((uint8_t *)sem_ptr, sizeof(VkSemaphore), + export_sem_free, ctx, 0); + if (!avbuf) { + av_free(sem_ptr); + return AVERROR(ENOMEM); + } + if (ff_vk_exec_add_dep_buf(ctx, exec, &avbuf, 1, 0) < 0) { + av_buffer_unref(&avbuf); + return AVERROR(ENOMEM); + } + *export_sem = VK_NULL_HANDLE; + return 0; +} + static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, const AVFrame *src, int flags) { @@ -4138,15 +4206,17 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, AVVulkanDeviceContext *hwctx = &p->p; FFVulkanFunctions *vk = &p->vkctx.vkfn; VulkanFramesPriv *fp = hwfc->hwctx; + const int planes = av_pix_fmt_count_planes(hwfc->sw_format); const int nb_images = ff_vk_count_images(f); VkImageDrmFormatModifierPropertiesEXT drm_mod = { .sType = VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_PROPERTIES_EXT, }; - VkSemaphoreWaitInfo wait_info = { - .sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, - .flags = 0x0, - .semaphoreCount = nb_images, - }; + const int nb_sems = nb_images; + + int sync_fd = -1; + VkSemaphore export_sem = VK_NULL_HANDLE; + int export_sem_submitted = 0; + FFVkExecContext *exec = NULL; AVDRMFrameDescriptor *drm_desc = av_mallocz(sizeof(*drm_desc)); if (!drm_desc) @@ -4156,11 +4226,95 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, if (err < 0) goto end; - /* Wait for the operation to finish so we can cleanly export it. */ - wait_info.pSemaphores = f->sem; - wait_info.pValues = f->sem_value; +#ifdef DMA_BUF_IOCTL_EXPORT_SYNC_FILE + if (p->vkctx.extensions & FF_VK_EXT_EXTERNAL_FD_SEM) { + VkSemaphoreSubmitInfo wait_infos[AV_NUM_DATA_POINTERS]; + VkSemaphoreSubmitInfo signal_info; + VkSemaphoreGetFdInfoKHR get_fd_info; + VkSemaphoreTypeCreateInfo sem_type_info = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO, + .semaphoreType = VK_SEMAPHORE_TYPE_BINARY, + }; + VkSemaphoreCreateInfo sem_spawn = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, + .pNext = &sem_type_info, + }; + VkExportSemaphoreCreateInfo ext_sem_info = { + .sType = VK_STRUCTURE_TYPE_EXPORT_SEMAPHORE_CREATE_INFO, + .handleTypes = VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_SYNC_FD_BIT, + }; + ff_vk_link_struct(&sem_type_info, &ext_sem_info); - vk->WaitSemaphores(hwctx->act_dev, &wait_info, UINT64_MAX); + ret = vk->CreateSemaphore(hwctx->act_dev, &sem_spawn, hwctx->alloc, &export_sem); + if (ret != VK_SUCCESS) { + av_log(hwctx, AV_LOG_ERROR, "Failed to create export semaphore: %s\n", + ff_vk_ret2str(ret)); + err = AVERROR_EXTERNAL; + goto sem_fail; + } + + exec = ff_vk_exec_get(&p->vkctx, &fp->export_exec); + vk->WaitForFences(hwctx->act_dev, 1, &exec->fence, VK_TRUE, UINT64_MAX); + ff_vk_exec_discard_deps(&p->vkctx, exec); + vk->ResetFences(hwctx->act_dev, 1, &exec->fence); + signal_info = (VkSemaphoreSubmitInfo) { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, + .semaphore = export_sem, + .stageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + }; + for (int i = 0; i < nb_sems; i++) { + wait_infos[i] = (VkSemaphoreSubmitInfo) { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, + .semaphore = f->sem[i], + .value = f->sem_value[i], + .stageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + }; + } + hwctx->lock_queue(hwfc->device_ctx, exec->qf, exec->qi); + ret = vk->QueueSubmit2(exec->queue, 1, &(VkSubmitInfo2) { + .sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, + .waitSemaphoreInfoCount = nb_sems, + .pWaitSemaphoreInfos = wait_infos, + .signalSemaphoreInfoCount = 1, + .pSignalSemaphoreInfos = &signal_info, + }, exec->fence); + hwctx->unlock_queue(hwfc->device_ctx, exec->qf, exec->qi); + if (ret != VK_SUCCESS) { + av_log(hwctx, AV_LOG_ERROR, "Failed to submit export semaphore signal: %s\n", + ff_vk_ret2str(ret)); + goto sem_fail; + } + exec->had_submission = 1; + export_sem_submitted = 1; + + get_fd_info = (VkSemaphoreGetFdInfoKHR) { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_GET_FD_INFO_KHR, + .semaphore = export_sem, + .handleType = VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_SYNC_FD_BIT, + }; + ret = vk->GetSemaphoreFdKHR(hwctx->act_dev, &get_fd_info, &sync_fd); + if (ret != VK_SUCCESS) { + av_log(hwctx, AV_LOG_ERROR, "Failed to get sync fd from export semaphore: %s\n", + ff_vk_ret2str(ret)); + err = AVERROR_EXTERNAL; + goto sem_fail; + } + err = add_export_sem_dep(&p->vkctx, exec, &export_sem); + if (err < 0) + goto sem_fail; + } else +#endif + + if (sync_fd < 0) { + VkSemaphoreWaitInfo wait_info = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, + .flags = 0x0, + .semaphoreCount = nb_sems, + .pSemaphores = f->sem, + .pValues = f->sem_value, + }; + vk->WaitSemaphores(hwctx->act_dev, &wait_info, UINT64_MAX); + } err = ff_hwframe_map_create(src->hw_frames_ctx, dst, src, &vulkan_unmap_to_drm, drm_desc); if (err < 0) @@ -4174,7 +4328,7 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, goto end; } - for (int i = 0; (i < nb_images) && (f->mem[i]); i++) { + for (int i = 0; (i < planes) && (f->mem[i]); i++) { VkMemoryGetFdInfoKHR export_info = { .sType = VK_STRUCTURE_TYPE_MEMORY_GET_FD_INFO_KHR, .memory = f->mem[i], @@ -4189,12 +4343,30 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, goto end; } +#if HAVE_LINUX_DMA_BUF_H && defined(DMA_BUF_IOCTL_IMPORT_SYNC_FILE) + if (sync_fd >= 0) { + int dup_fd = dup(sync_fd); + if (dup_fd >= 0) { + struct dma_buf_import_sync_file import_info = { + .flags = DMA_BUF_SYNC_WRITE, + .fd = dup_fd, + }; + if (ioctl(drm_desc->objects[i].fd, DMA_BUF_IOCTL_IMPORT_SYNC_FILE, &import_info) < 0) + av_log(hwfc, AV_LOG_WARNING, "DMA_BUF_IOCTL_IMPORT_SYNC_FILE failed: %s\n", av_err2str(AVERROR(errno))); + close(dup_fd); + } else { + av_log(hwfc, AV_LOG_WARNING, "dup(sync_fd) failed: %s\n", av_err2str(AVERROR(errno))); + } + } +#endif + drm_desc->nb_objects++; drm_desc->objects[i].size = f->size[i]; drm_desc->objects[i].format_modifier = drm_mod.drmFormatModifier; } - drm_desc->nb_layers = nb_images; + /* NV12 has 2 planes but 1 image/semaphore */ + drm_desc->nb_layers = FFMAX(planes, nb_images); for (int i = 0; i < drm_desc->nb_layers; i++) { VkFormat plane_vkfmt = av_vkfmt_from_pixfmt(hwfc->sw_format)[i]; @@ -4209,13 +4381,14 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, for (int j = 0; j < drm_desc->layers[i].nb_planes; j++) { VkSubresourceLayout layout; + int aspect_plane = (nb_images == 1) ? i : j; VkImageSubresource sub = { - .aspectMask = plane_index_to_aspect(j), + .aspectMask = plane_index_to_aspect(aspect_plane), }; drm_desc->layers[i].planes[j].object_index = FFMIN(i, drm_desc->nb_objects - 1); - vk->GetImageSubresourceLayout(hwctx->act_dev, f->img[i], &sub, &layout); + vk->GetImageSubresourceLayout(hwctx->act_dev, f->img[FFMIN(i, nb_images - 1)], &sub, &layout); drm_desc->layers[i].planes[j].offset = layout.offset; drm_desc->layers[i].planes[j].pitch = layout.rowPitch; } @@ -4235,13 +4408,30 @@ static int vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst, dst->width = src->width; dst->height = src->height; dst->data[0] = (uint8_t *)drm_desc; + dst->hw_frames_ctx = av_buffer_ref(src->hw_frames_ctx); + + if (sync_fd >= 0) + close(sync_fd); av_log(hwfc, AV_LOG_VERBOSE, "Mapped AVVkFrame to a DRM object!\n"); return 0; +sem_fail: + if (export_sem != VK_NULL_HANDLE) { + if (export_sem_submitted && exec) { + add_export_sem_dep(&p->vkctx, exec, &export_sem); + } + if (export_sem != VK_NULL_HANDLE && !export_sem_submitted) + vk->DestroySemaphore(hwctx->act_dev, export_sem, hwctx->alloc); + } end: + for (int i = 0; i < drm_desc->nb_objects; i++) { + close(drm_desc->objects[i].fd); + } av_free(drm_desc); + if (sync_fd >= 0) + close(sync_fd); return err; } -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
