PR #25651 opened by 5ym URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25651 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25651.patch
In CQP mode `av1_vaapi` never enabled CDEF and left `loop_filter_level[]` at 0, so the output had no deblocking and no CDEF. With bitrate control the driver fills these in, but in CQP mode `va_enc_av1.h` leaves them to the application, and iHD filters with the `filter_level` and `cdef_*` picture parameters. This sets them from `base_q_idx` in CQP mode only: - loop filter level `min((q - 32)^2 * 7 / 4096, 63)` for luma, half of it (rounded up) for chroma - CDEF with `cdef_bits = 2`, damping `3 + q / 64` and four candidates (including off), so the hardware can pick one per 64x64 block No options are added and other rate control modes are unchanged. The commit message explains the choice of values, why `support_cdef_channel_strength` is not checked (iHD reports 0 on DG2 but uses the strengths), and the effect on radeonsi and d3d12. On an Intel Arc A310, CQP needed 48% and 66% more bitrate than ICQ for the same XPSNR before this change; with it, the XPSNR BD-rate is -33.5% and -42.0%, close to ICQ. Details are in the commit message. >From e93f7bf29a28b38cac315a3ab74fcfa5b2519764 Mon Sep 17 00:00:00 2001 From: Ruk Doe <[email protected]> Date: Sun, 1 Nov 2026 15:13:43 +0900 Subject: [PATCH] avcodec/vaapi_encode_av1: enable loop filter and CDEF for CQP In CQP mode the encoder never enabled CDEF in the sequence header and left loop_filter_level[] at 0, so the output had no deblocking and no CDEF at all. With bitrate control the driver writes these parameters into the packed frame header at the bit offsets we pass. In CQP mode va_enc_av1.h says the offsets "should be set to 0 and ignored by driver", so the driver does not patch the packed header and the values are up to the application. The iHD driver filters with the filter_level and cdef_* picture parameters in that case. Choose them from base_q_idx: - loop_filter_level[0..1] = min((base_q_idx - 32)^2 * 7 / 4096, 63) (0 below 32), and half of that, rounded up, for U and V. The deblocking thresholds scale with the level, the blocking scales with the quantizer step, and the step grows faster than linearly with base_q_idx. GStreamer's vaav1enc also derives the level from the q index with a table; the quadratic stays within a few levels of its luma table and gives the same 15/8 luma/chroma levels at base_q_idx 128 (also the fixed values of the libva-utils av1encode sample). Sharpness stays 0. - CDEF with cdef_bits = 2, damping 3 + base_q_idx / 64 (as in GStreamer's vaav1enc) and four { primary strength, secondary strength code } candidates scaled by p = base_q_idx / 32: { p, 1 }, off, { p / 2, 1 } and { 2p, 2 }, i.e. secondary strength codes 1 and 2 (strengths 1 and 2). The hardware can choose one per 64x64 block, including turning CDEF off for detailed areas, as in the libva-utils and GStreamer encoders. Below base_q_idx 32 all strengths are 0. The parameters only depend on base_q_idx, so the per frame type offsets from i_qfactor/b_qfactor carry over to the filters. No options are added; the values only replace the previous all-off settings. Other rate control modes are unchanged. VAConfigAttribEncAV1.support_cdef_channel_strength ("Allows setting cdef_y_strengths and cdef_uv_strengths in PPS") is not checked: drivers read it differently and none of them needs it here. iHD reports 0 on DG2 but still programs the hardware with the strengths from the picture parameters as long as cdef_uv_strengths equals cdef_y_strengths (CheckCDEF()), and reports 1 from MTL on. radeonsi reports 1 only on VCN5, the only generation that takes explicit strengths. Mesa's d3d12 driver reports it when the D3D12 driver supports CDEF at all. GStreamer's vaav1enc reads the bit but does not use it, and passes equal luma and chroma strengths. Luma and chroma get the same strengths here, so DG2 accepts them; gating on the bit would drop CDEF on DG2, and bitrate control already enables CDEF without checking it. On radeonsi the firmware still writes the frame header and picks the loop filter level; CQP now enables CDEF there as well, with the strengths chosen by the firmware before VCN5 and the ones above on VCN5. As with bitrate control, a D3D12 driver without CDEF support now rejects the configuration in CQP mode too; VA-API has no capability bit for CDEF itself. Tested on an Intel Arc A310 (iHD driver) with a 1440x1080 interlaced broadcast recording, deinterlaced to 29.97 fps, two 60 s segments, CQP at qindex 70-120 and ICQ at global_quality 24-27. Quality is XPSNR as (6 * Y + U + V) / 8 of the per-plane values from the xpsnr filter. Before this change CQP needed 47.8% and 66.2% more bitrate than ICQ for the same XPSNR; changing b_qfactor did not help. Against the old CQP, the BD-rate is -33.5% and -42.0% (luma only: -30.5% and -37.4%), which brings CQP close to ICQ (+2.8% and +0.4%). The VMAF (v0.6.1) BD-rate is +1.1% and +1.4%; VMAF already rated the unfiltered CQP output above ICQ (-3.8% and -8.7%), so it does not reflect the blocking the filters remove. Encoding speed is unchanged (about 165-170 fps). ICQ used base_q_idx about 82/87/108 for I/P/B frames with luma loop filter levels about 4/5/10 (4/5/9 with this change) and cdef_y_strengths values (primary * 4 + secondary code) of 11-16. These numbers are from that one GPU only. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01XNTXZLNBnfhFULMqQoQmmY --- libavcodec/vaapi_encode_av1.c | 58 +++++++++++++++++++++++++++++++++-- 1 file changed, 55 insertions(+), 3 deletions(-) diff --git a/libavcodec/vaapi_encode_av1.c b/libavcodec/vaapi_encode_av1.c index b70f17580fb..df49303a3fc 100644 --- a/libavcodec/vaapi_encode_av1.c +++ b/libavcodec/vaapi_encode_av1.c @@ -446,11 +446,10 @@ static int vaapi_encode_av1_init_sequence_params(AVCodecContext *avctx) vseq->ip_period = base_ctx->b_per_p + 1; vseq->seq_fields.bits.enable_order_hint = sh->enable_order_hint; + vseq->seq_fields.bits.enable_cdef = sh->enable_cdef = 1; - if (!(ctx->va_rc_mode & VA_RC_CQP)) { + if (!(ctx->va_rc_mode & VA_RC_CQP)) vseq->bits_per_second = ctx->va_bit_rate; - vseq->seq_fields.bits.enable_cdef = sh->enable_cdef = 1; - } ret = vaapi_encode_av1_add_obu(avctx, obu, AV1_OBU_SEQUENCE_HEADER, &priv->sh); if (ret < 0) @@ -465,6 +464,56 @@ end: return ret; } +/** + * With VA_RC_CQP the driver does not patch the packed frame header, so + * choose the in-loop filter parameters here from base_q_idx and pass the + * same values in the picture parameters. In other modes the driver + * writes them into the header at the bit offsets we pass. + */ +static void vaapi_encode_av1_set_cqp_filters(AV1RawFrameHeader *fh, + VAEncPictureParameterBufferAV1 *vpic) +{ + const int q = fh->base_q_idx; + /* + * Deblocking thresholds scale with the filter level, and the + * blocking to remove scales with the quantizer step size, which + * grows faster than linearly with base_q_idx, so use a quadratic + * of base_q_idx. It reaches the maximum level 63 at 224. + */ + const int level = q > 32 ? FFMIN((q - 32) * (q - 32) * 7 >> 12, 63) : 0; + const int pri = q >> 5; + /* + * Four { primary strength, secondary strength code } CDEF candidates; + * the hardware picks one per 64x64 block. Put the main candidate + * first, and "off" keeps fine texture from being smoothed. + */ + const int cdef[4][2] = { + { pri, 1 }, + { 0, 0 }, + { pri >> 1, 1 }, + { pri << 1, 2 }, + }; + + fh->loop_filter_level[0] = vpic->filter_level[0] = level; + fh->loop_filter_level[1] = vpic->filter_level[1] = level; + fh->loop_filter_level[2] = vpic->filter_level_u = (level + 1) >> 1; + fh->loop_filter_level[3] = vpic->filter_level_v = (level + 1) >> 1; + + /* low base_q_idx: CDEF gains little, keep strengths at 0 */ + if (!pri) + return; + + fh->cdef_damping_minus_3 = vpic->cdef_damping_minus_3 = q >> 6; + fh->cdef_bits = vpic->cdef_bits = 2; + /* iHD rejects different luma and chroma CDEF strengths on DG2 */ + for (int i = 0; i < 4; i++) { + fh->cdef_y_pri_strength[i] = fh->cdef_uv_pri_strength[i] = cdef[i][0]; + fh->cdef_y_sec_strength[i] = fh->cdef_uv_sec_strength[i] = cdef[i][1]; + vpic->cdef_y_strengths[i] = vpic->cdef_uv_strengths[i] = + (cdef[i][0] << 2) | cdef[i][1]; + } +} + static int vaapi_encode_av1_init_picture_params(AVCodecContext *avctx, FFHWBaseEncodePicture *pic) { @@ -602,6 +651,9 @@ static int vaapi_encode_av1_init_picture_params(AVCodecContext *avctx, if (fh->frame_type == AV1_FRAME_KEY || fh->error_resilient_mode) fh->primary_ref_frame = AV1_PRIMARY_REF_NONE; + if (ctx->va_rc_mode & VA_RC_CQP) + vaapi_encode_av1_set_cqp_filters(fh, vpic); + vpic->base_qindex = fh->base_q_idx; vpic->frame_width_minus_1 = fh->frame_width_minus_1; vpic->frame_height_minus_1 = fh->frame_height_minus_1; -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
