Re: [FFmpeg-devel] [PATCH WIP v2 1/9] avfilter/dnn: Refactor DNN parameter configuration system

[Chen, Wenbin]

> -Original Message-
> From: ffmpeg-devel  On Behalf Of Zhao
> Zhili
> Sent: Sunday, April 28, 2024 2:47 PM
> To: ffmpeg-devel@ffmpeg.org
> Cc: Zhao Zhili 
> Subject: [FFmpeg-devel] [PATCH WIP v2 1/9] avfilter/dnn: Refactor DNN
> parameter configuration system
> 
> From: Zhao Zhili 
> +
> +void *ff_dnn_child_next(DnnContext *obj, void *prev) {
> +size_t pre_offset;
> +char *ptr;
> +
> +if (!prev) {
> +obj->clazz = &dnn_base_class;
> +return obj;
> +}
> +
> +pre_offset = (char *)prev - (char *)obj;
> +for (int i = 0; i < FF_ARRAY_ELEMS(dnn_backend_info_list) - 1; i++) {
> +if (dnn_backend_info_list[i].offset == pre_offset) {
> +ptr = (char *)obj + dnn_backend_info_list[i + 1].offset;
> +*(const AVClass **) ptr = dnn_backend_info_list[i + 1].class;
> +return ptr;
> +}
> +}
> +
> +return NULL;
> +}

Can this function be simplified by implementing AVFilter.preinit interface in 
each dnn filter
and assign class to DNNContext and TF/OV/THOption in preinit function?  I don't 
think setting AVClass
in child_next() function is the proper way.

wenbin

> +
> +const AVClass *ff_dnn_child_class_iterate(void **iter)
> +{
> +uintptr_t i = (uintptr_t) *iter;
> +
> +if (i < FF_ARRAY_ELEMS(dnn_backend_info_list)) {
> +*iter = (void *)(i + 1);
> +return dnn_backend_info_list[i].class;
> +}
> +
> +return NULL;
> +}
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH WIP 0/9] Refactor DNN

[Chen, Wenbin]

> > On Apr 29, 2024, at 18:29, Guo, Yejun 
> wrote:
> >
> >
> >
> >> -Original Message-
> >> From: ffmpeg-devel  On Behalf Of
> Zhao
> >> Zhili
> >> Sent: Sunday, April 28, 2024 6:55 PM
> >> To: FFmpeg development discussions and patches  >> de...@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH WIP 0/9] Refactor DNN
> >>
> >>
> >>
> >>> On Apr 28, 2024, at 18:34, Guo, Yejun  >> intel@ffmpeg.org> wrote:
> >>>
>  -Original Message-
>  From: ffmpeg-devel   > On Behalf Of Zhao Zhili
>  Sent: Sunday, April 28, 2024 12:42 AM
>  To: ffmpeg-devel@ffmpeg.org 
>  Cc: Zhao Zhili  >
>  Subject: [FFmpeg-devel] [PATCH WIP 0/9] Refactor DNN
> 
>  From: Zhao Zhili 
> 
>  During the refactor progress, I have found some serious issues, which
>  is not resolved by the patchset:
> 
>  1. Tensorflow backend is broken.
> 
>  I think it doesn't work since 2021 at least. For example, it destroy
>  a thread and create a new thread for each frame, and it destroy an
>  invalid thread at the first
>  frame:
> >>>
> >>> It works from the day that code is merged, till today. It is by design
> >>> to keep the code simplicity by using the feature that pthread_join
> >>> accepts a parameter that is not a joinable thread.
> >>>
> >>> Please share more info if you experienced a real case that it does not
> work.
> >>
> >> It will abort if ASSERT_LEVEL > 1.
> >>
> >> #define ASSERT_PTHREAD_ABORT(func, ret) do {\
> >>char errbuf[AV_ERROR_MAX_STRING_SIZE] = ""; \
> >>av_log(NULL, AV_LOG_FATAL, AV_STRINGIFY(func)   \
> >>   " failed with error: %s\n",  \
> >>   av_make_error_string(errbuf, AV_ERROR_MAX_STRING_SIZE,   \
> >>AVERROR(ret))); \
> >>abort();\
> >> } while (0)
> >>
> >> I think the check is there just to prevent call pthread_join(0, &ptr) by
> accident,
> >> so we shouldn’t do that on purpose.
> >>
> > Nice catch with configure assert level > 1, will fix, and patch also 
> > welcome,
> thanks.
> 
> If I read the code correctly, it destroy a thread and create a new thread for
> each
> frame. I think this “async” mode isn’t common in ffmpeg’s design. Create new
> thread
> for each frame can be heavy on some platforms. We use slice threading to
> improve
> parallel, and thread with command queue to improve throughput. In this
> case with
> tensorflow do the heavy lift, if it doesn’t support async operation, simple
> synchronous
> operation with tensorflow backend should be find. The “async” mode is
> unnecessary
> and use more resource  over the benefit it provides.

I think we need to keep async support.
1. Some model cannot make full use of resource. This may be caused by tensorflow
implementation or by model design. Async has benefit on this situation.
2. Async helps to build pipeline. You don't need to wait the output. If a 
"synchronous" filter
followed by another "synchronous" filter, it can be the bottle neck of the 
whole pipeline.

The benefit on these two situations will be more obvious if model is running on 
GPU.( Tensorflow
has not added device support yet.)

> 
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH WIP 0/9] Refactor DNN

[Chen, Wenbin]

> > On Apr 28, 2024, at 18:58, Paul B Mahol  wrote:
> >
> > Extremely low quality filters, both in source code quality and
> > performance/security and output quality should be queued for removal.

These filters cannot be removed because there are users using them.
What are your suggestions to improve them if you think they are low quality 
filters?

- Wenbin

> 
> I don’t think there is any replacement for the function provided by
> dnn_detect,
> which is very important for ROI encoding.
> 
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] doc: Add libtoch backend option to dnn_processing

[Chen, Wenbin]

> >> -Original Message-
> >> From: ffmpeg-devel  On Behalf Of
> >> wenbin.chen-at-intel@ffmpeg.org
> >> Sent: Monday, March 25, 2024 10:15 AM
> >> To: ffmpeg-devel@ffmpeg.org
> >> Subject: [FFmpeg-devel] [PATCH v2] doc: Add libtoch backend option to
> >> dnn_processing
> 
> Typo in commit message, you want libtorch, not libtoch.
> 
> - Leo Izen (Traneptora)

Thanks for your comment. I didn't notice that.
It is merged, so I cannot fix it.

> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] Changelog: Add libtorch

[Chen, Wenbin]

> On date Wednesday 2024-03-20 16:01:36 +0800, wenbin.chen-at-
> intel@ffmpeg.org wrote:
> > From: Wenbin Chen 
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  Changelog | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/Changelog b/Changelog
> > index e3ca52430c..d0c41887f3 100644
> > --- a/Changelog
> > +++ b/Changelog
> > @@ -35,6 +35,7 @@ version :
> >  - AEA muxer
> >  - ffmpeg CLI loopback decoders
> 
> >  - Support PacketTypeMetadata of PacketType in enhanced flv format
> 
> > +- Support libtorch as DNN backend
> 
> missing context, my take:
> libtorch backend in dnn filter
> 
> or:
> dnn filter libtorch backend
> 
> usually there is no capitalization and no verb in the items, but a
> description of the introduced feature

I update it in the new patch.

Thanks
Wenbin

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN backend

[Chen, Wenbin]

> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> > wenbin.chen-at-intel@ffmpeg.org
> > Sent: Monday, March 11, 2024 1:02 PM
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: [FFmpeg-devel] [PATCH v5] libavfi/dnn: add LibTorch as one of DNN
> > backend
> >
> > From: Wenbin Chen 
> >
> > PyTorch is an open source machine learning framework that accelerates
> > the path from research prototyping to production deployment. Official
> > website: https://pytorch.org/. We call the C++ library of PyTorch as
> > LibTorch, the same below.
> >
> > To build FFmpeg with LibTorch, please take following steps as reference:
> > 1. download LibTorch C++ library in 
> > https://pytorch.org/get-started/locally/,
> > please select C++/Java for language, and other options as your need.
> > Please download cxx11 ABI version (libtorch-cxx11-abi-shared-with-deps-
> > *.zip).
> > 2. unzip the file to your own dir, with command
> > unzip libtorch-shared-with-deps-latest.zip -d your_dir
> > 3. export libtorch_root/libtorch/include and
> > libtorch_root/libtorch/include/torch/csrc/api/include to $PATH
> > export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH
> > 4. config FFmpeg with ../configure --enable-libtorch --extra-cflag=-
> > I/libtorch_root/libtorch/include --extra-cflag=-
> > I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=-
> > L/libtorch_root/libtorch/lib/
> > 5. make
> >
> > To run FFmpeg DNN inference with LibTorch backend:
> > ./ffmpeg -i input.jpg -vf
> > dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y
> output.jpg
> > The LibTorch_model.pt can be generated by Python with torch.jit.script()
> api.
> > Please note, torch.jit.trace() is not recommanded, since it does not support
> > ambiguous input size.
> 
> Can you provide more detail (maybe a link from pytorch) about the
> libtorch_model.py generation and so we can have a try.
> 


This is a guide from pytorch: 
https://pytorch.org/tutorials/advanced/cpp_export.html
I will add it into commit log.

I didn't find a ready-made torchscript model to download. I'm afraid you'll 
have to export
the model yourself to test.

> >
> > Signed-off-by: Ting Fu 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  configure |   5 +-
> >  libavfilter/dnn/Makefile  |   1 +
> >  libavfilter/dnn/dnn_backend_torch.cpp | 597
> > ++
> >  libavfilter/dnn/dnn_interface.c   |   5 +
> >  libavfilter/dnn_filter_common.c   |  15 +-
> >  libavfilter/dnn_interface.h   |   2 +-
> >  libavfilter/vf_dnn_processing.c   |   3 +
> >  7 files changed, 624 insertions(+), 4 deletions(-)
> >  create mode 100644 libavfilter/dnn/dnn_backend_torch.cpp
> >
> > +static int fill_model_input_th(THModel *th_model, THRequestItem
> *request)
> > +{
> > +LastLevelTaskItem *lltask = NULL;
> > +TaskItem *task = NULL;
> > +THInferRequest *infer_request = NULL;
> > +DNNData input = { 0 };
> > +THContext *ctx = &th_model->ctx;
> > +int ret, width_idx, height_idx, channel_idx;
> > +
> > +lltask = (LastLevelTaskItem *)ff_queue_pop_front(th_model-
> > >lltask_queue);
> > +if (!lltask) {
> > +ret = AVERROR(EINVAL);
> > +goto err;
> > +}
> > +request->lltask = lltask;
> > +task = lltask->task;
> > +infer_request = request->infer_request;
> > +
> > +ret = get_input_th(th_model, &input, NULL);
> > +if ( ret != 0) {
> > +goto err;
> > +}
> > +width_idx = dnn_get_width_idx_by_layout(input.layout);
> > +height_idx = dnn_get_height_idx_by_layout(input.layout);
> > +channel_idx = dnn_get_channel_idx_by_layout(input.layout);
> > +input.dims[height_idx] = task->in_frame->height;
> > +input.dims[width_idx] = task->in_frame->width;
> > +input.data = av_malloc(input.dims[height_idx] * input.dims[width_idx] *
> > +   input.dims[channel_idx] * sizeof(float));
> > +if (!input.data)
> > +return AVERROR(ENOMEM);
> > +infer_request->input_tensor = new torch::Tensor();
> > +infer_request->output = new torch::Tensor();
> > +
> > +switch (th_model->model->func_type) {
> > +case DFT_PROCESS_FRAME:
> > +input.scale = 255;
> > +if (task->do_ioproc) {
> > +if (th_model->model->frame_pre_proc != NULL) {
> > +th_model->model->frame_pre_proc(task->in_frame, &input,
> > th_model->model->filter_ctx);
> > +} else {
> > +ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx);
> > +}
> > +}
> > +break;
> > +default:
> > +avpriv_report_missing_feature(NULL, "model function type %d",
> > th_model->model->func_type);
> > +break;
> > +}
> > +*infer_request->input_tensor = torch::from_blob(input.data,
> > +{1, 1, input.dims[channel_idx], input.dims[height_idx],
> > input.dims[width_idx]},
> 
> An extra dimension is added to support multiple fra

Re: [FFmpeg-devel] [PATCH v4] libavfi/dnn: add LibTorch as one of DNN backend

[Chen, Wenbin]

> 
> > On Feb 20, 2024, at 7:07 PM, wenbin.chen-at-intel@ffmpeg.org wrote:
> >
> > From: Wenbin Chen 
> >
> > PyTorch is an open source machine learning framework that accelerates
> > the path from research prototyping to production deployment. Official
> > website: https://pytorch.org/. We call the C++ library of PyTorch as
> > LibTorch, the same below.
> >
> > To build FFmpeg with LibTorch, please take following steps as reference:
> > 1. download LibTorch C++ library in 
> > https://pytorch.org/get-started/locally/,
> > please select C++/Java for language, and other options as your need.
> 
> I tested this locally with the current release (2.2.1) and this should 
> clarify that
> the cxx11 ABI version needs to be downloaded (libtorch-cxx11-abi-shared-
> with-deps-*.zip) otherwise it fails to link with undefined reference to 
> various
> c10 components.
> 
> 
> >
> > @@ -6886,6 +6888,7 @@ enabled libtensorflow && require libtensorflow
> tensorflow/c/c_api.h TF_Versi
> > enabled libtesseract  && require_pkg_config libtesseract tesseract
> tesseract/capi.h TessBaseAPICreate
> > enabled libtheora && require libtheora theora/theoraenc.h 
> > th_info_init
> -ltheoraenc -ltheoradec -logg
> > enabled libtls&& require_pkg_config libtls libtls tls.h 
> > tls_configure
> > +enabled libtorch  && check_cxxflags -std=c++14 && require_cpp
> libtorch torch/torch.h "torch::Tensor" -ltorch -lc10 -ltorch_cpu -lstdc++ -
> lpthread
> 
> This needs to be c++17 at least for the most recent (2.2.1) release. It fails 
> to
> compile with c++14.
> 
> - Cosmin

Thanks Cosmin

I will update them in patch v5.

-Wenbin
> 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavfi/dnn: add LibTorch as one of DNN backend

[Chen, Wenbin]

> Hello,
> 
> On Tue, 20 Feb 2024, at 05:48, wenbin.chen-at-intel@ffmpeg.org wrote:
> > From: Wenbin Chen 
> >
> > PyTorch is an open source machine learning framework that accelerates
> 
> OK for me
> 
> > the path from research prototyping to production deployment. Official
> > websit: https://pytorch.org/. We call the C++ library of PyTorch as
> 
> websitE

Fixed in Patch v4. Thanks

Wenbin

> 
> > LibTorch, the same below.
> >
> > To build FFmpeg with LibTorch, please take following steps as reference:
> > 1. download LibTorch C++ library in
> > https://pytorch.org/get-started/locally/,
> > please select C++/Java for language, and other options as your need.
> > 2. unzip the file to your own dir, with command
> > unzip libtorch-shared-with-deps-latest.zip -d your_dir
> > 3. export libtorch_root/libtorch/include and
> > libtorch_root/libtorch/include/torch/csrc/api/include to $PATH
> > export libtorch_root/libtorch/lib/ to $LD_LIBRARY_PATH
> > 4. config FFmpeg with ../configure --enable-libtorch
> > --extra-cflag=-I/libtorch_root/libtorch/include
> > --extra-cflag=-I/libtorch_root/libtorch/include/torch/csrc/api/include
> > --extra-ldflags=-L/libtorch_root/libtorch/lib/
> > 5. make
> >
> > To run FFmpeg DNN inference with LibTorch backend:
> > ./ffmpeg -i input.jpg -vf
> > dnn_processing=dnn_backend=torch:model=LibTorch_model.pt -y
> output.jpg
> > The LibTorch_model.pt can be generated by Python with
> > torch.jit.script() api. Please note, torch.jit.trace() is not
> > recommanded, since it does not support ambiguous input size.
> >
> > Signed-off-by: Ting Fu 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  configure |   5 +-
> >  libavfilter/dnn/Makefile  |   1 +
> >  libavfilter/dnn/dnn_backend_torch.cpp | 597
> ++
> >  libavfilter/dnn/dnn_interface.c   |   5 +
> >  libavfilter/dnn_filter_common.c   |  15 +-
> >  libavfilter/dnn_interface.h   |   2 +-
> >  libavfilter/vf_dnn_processing.c   |   3 +
> >  7 files changed, 624 insertions(+), 4 deletions(-)
> >  create mode 100644 libavfilter/dnn/dnn_backend_torch.cpp
> >
> > diff --git a/configure b/configure
> > index 2c635043dd..450ef54a80 100755
> > --- a/configure
> > +++ b/configure
> > @@ -279,6 +279,7 @@ External library support:
> >--enable-libtheora   enable Theora encoding via libtheora [no]
> >--enable-libtls  enable LibreSSL (via libtls), needed for
> > https support
> > if openssl, gnutls or mbedtls is not used
> > [no]
> > +  --enable-libtorchenable Torch as one DNN backend [no]
> >--enable-libtwolame  enable MP2 encoding via libtwolame [no]
> >--enable-libuavs3d   enable AVS3 decoding via libuavs3d [no]
> >--enable-libv4l2 enable libv4l2/v4l-utils [no]
> > @@ -1901,6 +1902,7 @@ EXTERNAL_LIBRARY_LIST="
> >  libtensorflow
> >  libtesseract
> >  libtheora
> > +libtorch
> >  libtwolame
> >  libuavs3d
> >  libv4l2
> > @@ -2781,7 +2783,7 @@ cbs_vp9_select="cbs"
> >  deflate_wrapper_deps="zlib"
> >  dirac_parse_select="golomb"
> >  dovi_rpu_select="golomb"
> > -dnn_suggest="libtensorflow libopenvino"
> > +dnn_suggest="libtensorflow libopenvino libtorch"
> >  dnn_deps="avformat swscale"
> >  error_resilience_select="me_cmp"
> >  evcparse_select="golomb"
> > @@ -6886,6 +6888,7 @@ enabled libtensorflow && require
> > libtensorflow tensorflow/c/c_api.h TF_Versi
> >  enabled libtesseract  && require_pkg_config libtesseract tesseract
> > tesseract/capi.h TessBaseAPICreate
> >  enabled libtheora && require libtheora theora/theoraenc.h
> > th_info_init -ltheoraenc -ltheoradec -logg
> >  enabled libtls&& require_pkg_config libtls libtls tls.h
> > tls_configure
> > +enabled libtorch  && check_cxxflags -std=c++14 && require_cpp
> > libtorch torch/torch.h "torch::Tensor" -ltorch -lc10 -ltorch_cpu
> > -lstdc++ -lpthread
> >  enabled libtwolame&& require libtwolame twolame.h twolame_init
> > -ltwolame &&
> >   { check_lib libtwolame twolame.h
> > twolame_encode_buffer_float32_interleaved -ltwolame ||
> > die "ERROR: libtwolame must be
> > installed and version must be >= 0.3.10"; }
> > diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
> > index 5d5697ea42..3d09927c98 100644
> > --- a/libavfilter/dnn/Makefile
> > +++ b/libavfilter/dnn/Makefile
> > @@ -6,5 +6,6 @@ OBJS-$(CONFIG_DNN)   +=
> > dnn/dnn_backend_common.o
> >
> >  DNN-OBJS-$(CONFIG_LIBTENSORFLOW) += dnn/dnn_backend_tf.o
> >  DNN-OBJS-$(CONFIG_LIBOPENVINO)   +=
> dnn/dnn_backend_openvino.o
> > +DNN-OBJS-$(CONFIG_LIBTORCH)  += dnn/dnn_backend_torch.o
> >
> >  OBJS-$(CONFIG_DNN)   += $(DNN-OBJS-yes)
> > diff --git a/libavfilter/dnn/dnn_backend_torch.cpp
> > b/libavfilter/dnn/dnn_backend_torch.cpp
> > new fil

Re: [FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend

[Chen, Wenbin]

> > Hello,
> >
> > On Fri, 2 Feb 2024, at 08:26, wenbin.chen-at-intel@ffmpeg.org wrote:
> > > +static void infer_completion_callback(void *args) {
> > > +THRequestItem *request = (THRequestItem*)args;
> > > +LastLevelTaskItem *lltask = request->lltask;
> > > +TaskItem *task = lltask->task;
> > > +DNNData outputs = { 0 };
> > > +THInferRequest *infer_request = request->infer_request;
> > > +THModel *th_model = (THModel *)task->model;
> > > +torch::Tensor *output = infer_request->output;
> > > +
> > > +c10::IntArrayRef sizes = output->sizes();
> > > +assert(sizes.size == 5);
> >
> > Why 5?
> 
> 5 means 5 channels: [batch_size, frame_number, channel, height, width]

Sorry, I mean 5 dimensions.

> I only add video SR support, so it only support this type of data for now.
> I will change the code to be more easy to read.
> 
> >
> > > +outputs.order = DCO_RGB;
> > > +outputs.layout = DL_NCHW;
> > > +outputs.dims[2] = sizes.at(3);
> > > +outputs.dims[3] = sizes.at(4);
> > > +outputs.dt = DNN_FLOAT;
> > > +outputs.dims[1] = 3;
> >
> > Why 3?
> 
> It is RGB so the channel is 3, but I should use sizes.at(2) instead of a magic
> number.
> Thanks for pointing it out. I will update it in patch v3.
> 
> >
> >
> > --
> > Jean-Baptiste Kempf -  President
> > +33 672 704 734
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] libavfi/dnn: add LibTorch as one of DNN backend

[Chen, Wenbin]

> Hello,
> 
> On Fri, 2 Feb 2024, at 08:26, wenbin.chen-at-intel@ffmpeg.org wrote:
> > +static void infer_completion_callback(void *args) {
> > +THRequestItem *request = (THRequestItem*)args;
> > +LastLevelTaskItem *lltask = request->lltask;
> > +TaskItem *task = lltask->task;
> > +DNNData outputs = { 0 };
> > +THInferRequest *infer_request = request->infer_request;
> > +THModel *th_model = (THModel *)task->model;
> > +torch::Tensor *output = infer_request->output;
> > +
> > +c10::IntArrayRef sizes = output->sizes();
> > +assert(sizes.size == 5);
> 
> Why 5?

5 means 5 channels: [batch_size, frame_number, channel, height, width]
I only add video SR support, so it only support this type of data for now.
I will change the code to be more easy to read.

> 
> > +outputs.order = DCO_RGB;
> > +outputs.layout = DL_NCHW;
> > +outputs.dims[2] = sizes.at(3);
> > +outputs.dims[3] = sizes.at(4);
> > +outputs.dt = DNN_FLOAT;
> > +outputs.dims[1] = 3;
> 
> Why 3?

It is RGB so the channel is 3, but I should use sizes.at(2) instead of a magic 
number.
Thanks for pointing it out. I will update it in patch v3.

> 
> 
> --
> Jean-Baptiste Kempf -  President
> +33 672 704 734
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] swscale: don't assign range converters for float

[Chen, Wenbin]

> On Mon, 27 Nov 2023 02:10:11 +0000 "Chen, Wenbin"  intel@ffmpeg.org> wrote:
> > > > From: Niklas Haas 
> > > >
> > > > This logic was incongruent with logic used elsewhere, where floating
> > > > point formats are explicitly exempted from range conversion. Fixes an
> > > > issue where floating point formats were not going through special
> > > > unscaled converters even when it was otherwise possible.
> > > > ---
> > > >  libswscale/swscale.c | 3 ++-
> > > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/libswscale/swscale.c b/libswscale/swscale.c
> > > > index 46ba68fe6a..a66db22767 100644
> > > > --- a/libswscale/swscale.c
> > > > +++ b/libswscale/swscale.c
> > > > @@ -534,7 +534,8 @@ av_cold void
> ff_sws_init_range_convert(SwsContext
> > > > *c)
> > > >  {
> > > >  c->lumConvertRange = NULL;
> > > >  c->chrConvertRange = NULL;
> > > > -if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat)) {
> > > > +if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat) &&
> > > > +!isFloat(c->srcFormat) && !isFloat(c->dstFormat)) {
> > > >  if (c->dstBpc <= 14) {
> > > >  if (c->srcRange) {
> > > >  c->lumConvertRange = lumRangeFromJpeg_c;
> > > > --
> > > > 2.42.0
> > > >
> > >
> > > This patchset works for me. Thanks for your quick fixing.
> >
> > Ping. When can this patchset be merged?
> 
> Hi, do you still need this patch, now that YUV negotiation means
> vf_scale no longer changes the filter range dynamically?

The problem is not solved.
Command " ffmpeg -i input.png -vf format=grayf32,format=gray8 output.png " 
still reports
error: "Assertion c->srcBpc == 16 failed at libswscale/x86/swscale.c:533"
After I add --disable-sse2, the problem is unseen.

> 
> >
> > Thanks
> > Wenbin
> > >
> > > > ___
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel@ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] swscale: don't assign range converters for float

[Chen, Wenbin]

> > > From: Niklas Haas 
> > >
> > > This logic was incongruent with logic used elsewhere, where floating
> > > point formats are explicitly exempted from range conversion. Fixes an
> > > issue where floating point formats were not going through special
> > > unscaled converters even when it was otherwise possible.
> > > ---
> > >  libswscale/swscale.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/libswscale/swscale.c b/libswscale/swscale.c
> > > index 46ba68fe6a..a66db22767 100644
> > > --- a/libswscale/swscale.c
> > > +++ b/libswscale/swscale.c
> > > @@ -534,7 +534,8 @@ av_cold void
> ff_sws_init_range_convert(SwsContext
> > > *c)
> > >  {
> > >  c->lumConvertRange = NULL;
> > >  c->chrConvertRange = NULL;
> > > -if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat)) {
> > > +if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat) &&
> > > +!isFloat(c->srcFormat) && !isFloat(c->dstFormat)) {
> > >  if (c->dstBpc <= 14) {
> > >  if (c->srcRange) {
> > >  c->lumConvertRange = lumRangeFromJpeg_c;
> > > --
> > > 2.42.0
> > >
> >
> > This patchset works for me. Thanks for your quick fixing.
> 
> Ping. When can this patchset be merged?
> 
> Thanks
> Wenbin

Anyone who can help to merge this patchset?

Thanks
Wenbin

> >
> > > ___
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] swscale: don't assign range converters for float

[Chen, Wenbin]

> > From: Niklas Haas 
> >
> > This logic was incongruent with logic used elsewhere, where floating
> > point formats are explicitly exempted from range conversion. Fixes an
> > issue where floating point formats were not going through special
> > unscaled converters even when it was otherwise possible.
> > ---
> >  libswscale/swscale.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/libswscale/swscale.c b/libswscale/swscale.c
> > index 46ba68fe6a..a66db22767 100644
> > --- a/libswscale/swscale.c
> > +++ b/libswscale/swscale.c
> > @@ -534,7 +534,8 @@ av_cold void ff_sws_init_range_convert(SwsContext
> > *c)
> >  {
> >  c->lumConvertRange = NULL;
> >  c->chrConvertRange = NULL;
> > -if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat)) {
> > +if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat) &&
> > +!isFloat(c->srcFormat) && !isFloat(c->dstFormat)) {
> >  if (c->dstBpc <= 14) {
> >  if (c->srcRange) {
> >  c->lumConvertRange = lumRangeFromJpeg_c;
> > --
> > 2.42.0
> >
> 
> This patchset works for me. Thanks for your quick fixing.

Ping. When can this patchset be merged?

Thanks
Wenbin 
> 
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] swscale: don't assign range converters for float

[Chen, Wenbin]

> From: Niklas Haas 
> 
> This logic was incongruent with logic used elsewhere, where floating
> point formats are explicitly exempted from range conversion. Fixes an
> issue where floating point formats were not going through special
> unscaled converters even when it was otherwise possible.
> ---
>  libswscale/swscale.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/libswscale/swscale.c b/libswscale/swscale.c
> index 46ba68fe6a..a66db22767 100644
> --- a/libswscale/swscale.c
> +++ b/libswscale/swscale.c
> @@ -534,7 +534,8 @@ av_cold void ff_sws_init_range_convert(SwsContext
> *c)
>  {
>  c->lumConvertRange = NULL;
>  c->chrConvertRange = NULL;
> -if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat)) {
> +if (c->srcRange != c->dstRange && !isAnyRGB(c->dstFormat) &&
> +!isFloat(c->srcFormat) && !isFloat(c->dstFormat)) {
>  if (c->dstBpc <= 14) {
>  if (c->srcRange) {
>  c->lumConvertRange = lumRangeFromJpeg_c;
> --
> 2.42.0
> 

This patchset works for me. Thanks for your quick fixing.

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3 1/8] swscale: fix sws_setColorspaceDetails after sws_init_context

[Chen, Wenbin]

> 
> Will apply soon.
>
Hi Niklas:

This patchset causes a regression.
The command: "ffmpeg -i input.png -vf format=grayf32,format=gray8 output.png" 
reports error.
If I configure with "--disable-sse2", the error is unseen.

Thanks
Wenbin
 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 3/3] libavfilter/dnn: Initialze DNNData variables

[Chen, Wenbin]

> > On Sep 20, 2023, at 10:26, wenbin.chen-at-intel@ffmpeg.org wrote:
> >
> > From: Wenbin Chen 
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> > libavfilter/dnn/dnn_backend_tf.c | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavfilter/dnn/dnn_backend_tf.c
> b/libavfilter/dnn/dnn_backend_tf.c
> > index b521de7fbe..e1e8cef0d2 100644
> > --- a/libavfilter/dnn/dnn_backend_tf.c
> > +++ b/libavfilter/dnn/dnn_backend_tf.c
> > @@ -629,6 +629,7 @@ static int fill_model_input_tf(TFModel *tf_model,
> TFRequestItem *request) {
> > TFContext *ctx = &tf_model->ctx;
> > int ret = 0;
> >
> > +memset(&input, 0, sizeof(input));
> 
> Can be simplified with DNNData input = { 0 };

Thanks for your advice. I update it in patch v2. 

> 
> > lltask = ff_queue_pop_front(tf_model->lltask_queue);
> > av_assert0(lltask);
> > task = lltask->task;
> > @@ -724,7 +725,7 @@ static void infer_completion_callback(void *args) {
> > TFModel *tf_model = task->model;
> > TFContext *ctx = &tf_model->ctx;
> >
> > -outputs = av_malloc_array(task->nb_output, sizeof(*outputs));
> > +outputs = av_calloc(task->nb_output, sizeof(*outputs));
> > if (!outputs) {
> > av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for
> *outputs\n");
> > goto err;
> > --
> > 2.34.1
> >
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Flush cached frames before reset encoder

[Chen, Wenbin]

> wenbin.chen-at-intel@ffmpeg.org:
> > From: Wenbin Chen 
> >
> > According to https://github.com/Intel-Media-
> SDK/MediaSDK/blob/master/doc/mediasdk-man.md#configuration-change.
> > Before calling MFXVideoENCODE_Reset, The application needs to retrieve
> > any cached frames in the SDK encoder.
> > A loop is added before MFXVideoENCODE_Reset to retrieve cached frames
> > and add them to async_fifo, so that dynamic configuration works when
> > async_depth > 1.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/qsvenc.c | 118 +++-
> >  1 file changed, 63 insertions(+), 55 deletions(-)
> >
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index 2f0e94a914..f3b488dec8 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -1600,7 +1600,7 @@ int ff_qsv_enc_init(AVCodecContext *avctx,
> QSVEncContext *q)
> >
> >  q->param.AsyncDepth = q->async_depth;
> >
> > -q->async_fifo = av_fifo_alloc2(q->async_depth, sizeof(QSVPacket), 0);
> > +q->async_fifo = av_fifo_alloc2(q->async_depth, sizeof(QSVPacket),
> AV_FIFO_FLAG_AUTO_GROW);
> 
> If you use AV_FIFO_FLAG_AUTO_GROW, you need to handle av_fifo_write()
> errors.

Thanks. I will fix it in patch v2.

> 
> >  if (!q->async_fifo)
> >  return AVERROR(ENOMEM);
> >
> > @@ -2296,58 +2296,6 @@ static int
> update_pic_timing_sei(AVCodecContext *avctx, QSVEncContext *q)
> >  return updated;
> >  }
> >
> > -static int update_parameters(AVCodecContext *avctx, QSVEncContext *q,
> > - const AVFrame *frame)
> > -{
> > -int needReset = 0, ret = 0;
> > -
> > -if (!frame || avctx->codec_id == AV_CODEC_ID_MJPEG)
> > -return 0;
> > -
> > -needReset = update_qp(avctx, q);
> > -needReset |= update_max_frame_size(avctx, q);
> > -needReset |= update_gop_size(avctx, q);
> > -needReset |= update_rir(avctx, q);
> > -needReset |= update_low_delay_brc(avctx, q);
> > -needReset |= update_frame_rate(avctx, q);
> > -needReset |= update_bitrate(avctx, q);
> > -needReset |= update_pic_timing_sei(avctx, q);
> > -ret = update_min_max_qp(avctx, q);
> > -if (ret < 0)
> > -return ret;
> > -needReset |= ret;
> > -if (!needReset)
> > -return 0;
> > -
> > -if (avctx->hwaccel_context) {
> > -AVQSVContext *qsv = avctx->hwaccel_context;
> > -int i, j;
> > -q->param.ExtParam = q->extparam;
> > -for (i = 0; i < qsv->nb_ext_buffers; i++)
> > -q->param.ExtParam[i] = qsv->ext_buffers[i];
> > -q->param.NumExtParam = qsv->nb_ext_buffers;
> > -
> > -for (i = 0; i < q->nb_extparam_internal; i++) {
> > -for (j = 0; j < qsv->nb_ext_buffers; j++) {
> > -if (qsv->ext_buffers[j]->BufferId == 
> > q->extparam_internal[i]-
> >BufferId)
> > -break;
> > -}
> > -if (j < qsv->nb_ext_buffers)
> > -continue;
> > -q->param.ExtParam[q->param.NumExtParam++] = q-
> >extparam_internal[i];
> > -}
> > -} else {
> > -q->param.ExtParam= q->extparam_internal;
> > -q->param.NumExtParam = q->nb_extparam_internal;
> > -}
> > -av_log(avctx, AV_LOG_DEBUG, "Parameter change, call msdk reset.\n");
> > -ret = MFXVideoENCODE_Reset(q->session, &q->param);
> > -if (ret < 0)
> > -return ff_qsv_print_error(avctx, ret, "Error during resetting");
> > -
> > -return 0;
> > -}
> > -
> >  static int encode_frame(AVCodecContext *avctx, QSVEncContext *q,
> >  const AVFrame *frame)
> >  {
> > @@ -2438,7 +2386,7 @@ static int encode_frame(AVCodecContext *avctx,
> QSVEncContext *q,
> >
> >  if (ret < 0) {
> >  ret = (ret == MFX_ERR_MORE_DATA) ?
> > -   0 : ff_qsv_print_error(avctx, ret, "Error during encoding");
> > +   AVERROR(EAGAIN) : ff_qsv_print_error(avctx, ret, "Error 
> > during
> encoding");
> >  goto free;
> >  }
> >
> > @@ -2466,6 +2414,66 @@ nomem:
> >  goto free;
> >  }
> >
> > +static int update_parameters(AVCodecContext *avctx, QSVEncContext *q,
> > + const AVFrame *frame)
> > +{
> > +int needReset = 0, ret = 0;
> > +
> > +if (!frame || avctx->codec_id == AV_CODEC_ID_MJPEG)
> > +return 0;
> > +
> > +needReset = update_qp(avctx, q);
> > +needReset |= update_max_frame_size(avctx, q);
> > +needReset |= update_gop_size(avctx, q);
> > +needReset |= update_rir(avctx, q);
> > +needReset |= update_low_delay_brc(avctx, q);
> > +needReset |= update_frame_rate(avctx, q);
> > +needReset |= update_bitrate(avctx, q);
> > +needReset |= update_pic_timing_sei(avctx, q);
> > +ret = update_min_max_qp(avctx, q);
> > +if (ret < 0)
> > +return ret;
> > +needReset |= ret;
> > +if (!needReset)
> > +return 0;
> > +
> > +if (avctx->hwaccel_context) {
> > +

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Do not pass RGB solorspace to VPL/MSDK

[Chen, Wenbin]

> From: Wenbin Chen 
> 
> When encode RGB frame, Intel driver convert RGB to YUV, so we cannot
> set RGB colorspace to VPL/MSDK.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/qsvenc.c | 7 ++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> index 2f0e94a914..944a76f4f1 100644
> --- a/libavcodec/qsvenc.c
> +++ b/libavcodec/qsvenc.c
> @@ -1185,7 +1185,12 @@ static int init_video_param(AVCodecContext
> *avctx, QSVEncContext *q)
>  q->extvsi.ColourDescriptionPresent = 1;
>  q->extvsi.ColourPrimaries = avctx->color_primaries;
>  q->extvsi.TransferCharacteristics = avctx->color_trc;
> -q->extvsi.MatrixCoefficients = avctx->colorspace;
> +if (avctx->colorspace == AVCOL_SPC_RGB)
> +//YUV will be converted to RGB, so RGB colorspace is not 
> supported

Comment is wrong. It should be " RGB will be converted to YUV".  I will send 
patch v2.

> +q->extvsi.MatrixCoefficients = AVCOL_SPC_UNSPECIFIED;
> +else
> +q->extvsi.MatrixCoefficients = avctx->colorspace;
> +
>  }
> 
>  if ((avctx->codec_id != AV_CODEC_ID_VP9) && (q->extvsi.VideoFullRange
> || q->extvsi.ColourDescriptionPresent)) {
> --
> 2.34.1
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] libavfilter/qsvvpp: Use different alignment for YUV420P format

[Chen, Wenbin]

> On Do, 2022-12-01 at 10:48 +0800, wenbin.chen-at-intel@ffmpeg.org
> wrote:
> > From: Wenbin Chen 
> >
> > When process yuv420 frames, FFmpeg uses same alignment on Y/U/V
> > planes. VPL and MSDK use Y plane's pitch / 2 as U/V planes's
> > pitch, which makes U/V planes 16-bytes aligned. We need to set
> > a separate alignment to meet runtime's behaviour.
> >
> > Now alignment is changed to 16 so that the linesizes of U/V planes
> > meet the requirment of VPL/MSDK. Add get_buffer.video callback to
> > qsv filters to change the default get_buffer behaviour.
> >
> > Now the commandline works fine:
> > ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 3082x1884 \
> > -i ./3082x1884.yuv -vf 'vpp_qsv=w=2466:h=1508' -f rawvideo \
> > -pix_fmt yuv420p 2466_1508.yuv
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavfilter/qsvvpp.c | 13 +
> >  libavfilter/qsvvpp.h |  1 +
> >  libavfilter/vf_deinterlace_qsv.c |  1 +
> >  libavfilter/vf_overlay_qsv.c |  2 ++
> >  libavfilter/vf_scale_qsv.c   |  1 +
> >  libavfilter/vf_vpp_qsv.c |  1 +
> >  6 files changed, 19 insertions(+)
> >
> > diff --git a/libavfilter/qsvvpp.c b/libavfilter/qsvvpp.c
> > index 8428ee89ab..d5cfeab402 100644
> > --- a/libavfilter/qsvvpp.c
> > +++ b/libavfilter/qsvvpp.c
> > @@ -1003,3 +1003,16 @@ int ff_qsvvpp_create_mfx_session(void *ctx,
> >  }
> >
> >  #endif
> > +
> > +AVFrame *ff_qsvvpp_get_video_buffer(AVFilterLink *inlink, int w, int h)
> > +{
> > +/* When process YUV420 frames, FFmpeg uses same alignment on
> Y/U/V
> > + * planes. VPL and MSDK use Y plane's pitch / 2 as U/V planes's
> > + * pitch, which makes U/V planes 16-bytes aligned. We need to set a
> > + * separate alignment to meet runtime's behaviour.
> > +*/
> > +return ff_default_get_video_buffer2(inlink,
> > +FFALIGN(inlink->w, 32),
> > +FFALIGN(inlink->h, 32),
> > +16);
> 
> It uses the same alignment for all formats, which is conflicted with the 
> title.
> 
> Thanks
> Haihao
> 
I will change the title and send the patch again.

Thanks
Wenbin
> 
> > +}
> > diff --git a/libavfilter/qsvvpp.h b/libavfilter/qsvvpp.h
> > index a8cfcc565a..6f7c9bfc15 100644
> > --- a/libavfilter/qsvvpp.h
> > +++ b/libavfilter/qsvvpp.h
> > @@ -127,4 +127,5 @@ int ff_qsvvpp_print_warning(void *log_ctx,
> mfxStatus err,
> >  int ff_qsvvpp_create_mfx_session(void *ctx, void *loader, mfxIMPL
> > implementation,
> >   mfxVersion *pver, mfxSession *psession);
> >
> > +AVFrame *ff_qsvvpp_get_video_buffer(AVFilterLink *inlink, int w, int h);
> >  #endif /* AVFILTER_QSVVPP_H */
> > diff --git a/libavfilter/vf_deinterlace_qsv.c
> > b/libavfilter/vf_deinterlace_qsv.c
> > index 98ed7283ad..6c94923f02 100644
> > --- a/libavfilter/vf_deinterlace_qsv.c
> > +++ b/libavfilter/vf_deinterlace_qsv.c
> > @@ -581,6 +581,7 @@ static const AVFilterPad qsvdeint_inputs[] = {
> >  .name = "default",
> >  .type = AVMEDIA_TYPE_VIDEO,
> >  .filter_frame = qsvdeint_filter_frame,
> > +.get_buffer.video = ff_qsvvpp_get_video_buffer,
> >  },
> >  };
> >
> > diff --git a/libavfilter/vf_overlay_qsv.c b/libavfilter/vf_overlay_qsv.c
> > index d947a1faa1..1a2c1b1e96 100644
> > --- a/libavfilter/vf_overlay_qsv.c
> > +++ b/libavfilter/vf_overlay_qsv.c
> > @@ -399,11 +399,13 @@ static const AVFilterPad overlay_qsv_inputs[] = {
> >  .name  = "main",
> >  .type  = AVMEDIA_TYPE_VIDEO,
> >  .config_props  = config_main_input,
> > +.get_buffer.video = ff_qsvvpp_get_video_buffer,
> >  },
> >  {
> >  .name  = "overlay",
> >  .type  = AVMEDIA_TYPE_VIDEO,
> >  .config_props  = config_overlay_input,
> > +.get_buffer.video = ff_qsvvpp_get_video_buffer,
> >  },
> >  };
> >
> > diff --git a/libavfilter/vf_scale_qsv.c b/libavfilter/vf_scale_qsv.c
> > index 758e730f78..36d5f3a6ec 100644
> > --- a/libavfilter/vf_scale_qsv.c
> > +++ b/libavfilter/vf_scale_qsv.c
> > @@ -641,6 +641,7 @@ static const AVFilterPad qsvscale_inputs[] = {
> >  .name = "default",
> >  .type = AVMEDIA_TYPE_VIDEO,
> >  .filter_frame = qsvscale_filter_frame,
> > +.get_buffer.video = ff_qsvvpp_get_video_buffer,
> >  },
> >  };
> >
> > diff --git a/libavfilter/vf_vpp_qsv.c b/libavfilter/vf_vpp_qsv.c
> > index 4a053f9145..b26d19c3bc 100644
> > --- a/libavfilter/vf_vpp_qsv.c
> > +++ b/libavfilter/vf_vpp_qsv.c
> > @@ -634,6 +634,7 @@ static const AVFilterPad vpp_inputs[] = {
> >  .name  = "default",
> >  .type  = AVMEDIA_TYPE_VIDEO,
> >  .config_props  = config_input,
> > +.get_buffer.video = ff_qsvvpp_get_video_buffer,
> >  },
> >  };
> >
___
ffmpeg-devel mailing list

Re: [FFmpeg-devel] [PATCH] libavfilter/qsvvpp: Use different alignment for YUV420P format

[Chen, Wenbin]

> On 11/29/2022 5:07 AM, wenbin.chen-at-intel@ffmpeg.org wrote:
> > From: Wenbin Chen 
> >
> > When process yuv420 frames, FFmpeg use same alignment on Y/U/V
> > planes. VPL and MSDK use Y plane's pitch / 2 as U/V planes's
> > pitch, which make U/V planes 16-bytes aligned. We need to set
> > a separate alignment to meet runtime's behaviour.
> >
> > Now the commandline works fine:
> > ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 3082x1884 \
> > -i ./3082x1884.yuv -vf 'vpp_qsv=w=2466:h=1508' -f rawvideo \
> > -pix_fmt yuv420p 2466_1508.yuv
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >   libavfilter/qsvvpp.c | 12 +---
> >   1 file changed, 9 insertions(+), 3 deletions(-)
> >
> > diff --git a/libavfilter/qsvvpp.c b/libavfilter/qsvvpp.c
> > index 8428ee89ab..ad09114cb7 100644
> > --- a/libavfilter/qsvvpp.c
> > +++ b/libavfilter/qsvvpp.c
> > @@ -408,9 +408,15 @@ static QSVFrame *submit_frame(QSVVPPContext
> *s, AVFilterLink *inlink, AVFrame *p
> >   } else {
> >   /* make a copy if the input is not padded as libmfx requires */
> >   if (picref->height & 31 || picref->linesize[0] & 31) {
> > -qsv_frame->frame = ff_get_video_buffer(inlink,
> > -   FFALIGN(inlink->w, 32),
> > -   FFALIGN(inlink->h, 32));
> > +/* When process YUV420 frames, FFmpeg uses same alignment on
> Y/U/V
> > + * planes. VPL and MSDK use Y plane's pitch / 2 as U/V planes's
> > + * pitch, which makes U/V planes 16-bytes aligned. We need to 
> > set
> a
> > + * separate alignment to meet runtime's behaviour.
> > + */
> > +qsv_frame->frame = ff_default_get_video_buffer2(inlink,
> 
> I think the proper way to do this is by setting a custom AVFilterPad
> get_buffer.video() callback, which will be called instead of
> ff_default_get_video_buffer() (Which uses an automatically chosen
> alignment at runtime) by ff_get_video_buffer().
> 
> See other filters like vf_pad, vf_transpose_vaapi, etc.

Thanks for your advice. I will try it.
> 
> > +FFALIGN(inlink->w, 32),
> > +FFALIGN(inlink->h, 32),
> > +16);
> >   if (!qsv_frame->frame)
> >   return NULL;
> >
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] lavc/cbs_av1: restore CodedBitstreamAV1Context when AVERROR(ENOSPC)

[Chen, Wenbin]

> From: Haihao Xiang 
> 
> The current pbc might be small for an obu frame, so a new pbc is
> required then parse this obu frame again. Because
> CodedBitstreamAV1Context has already been updated for this obu frame,
> we
> need to restore CodedBitstreamAV1Context, otherwise
> CodedBitstreamAV1Context doesn't match this obu frame when parsing obu
> frame again, e.g. CodedBitstreamAV1Context.order_hint.
> 
> $ ffmpeg -i input.ivf -c:v copy -f null -
> [...]
> [av1_frame_merge @ 0x558bc3d6f880] ref_order_hint[i] does not match
> inferred value: 20, but should be 22.
> [av1_frame_merge @ 0x558bc3d6f880] Failed to write unit 1 (type 6).
> [av1_frame_merge @ 0x558bc3d6f880] Failed to write packet.
> [obu @ 0x558bc3d6e040] av1_frame_merge filter failed to send output
> packet
> ---
>  libavcodec/cbs_av1.c | 85 ++--
>  1 file changed, 67 insertions(+), 18 deletions(-)
> 
> diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
> index 154d9156cf..585bc72cdb 100644
> --- a/libavcodec/cbs_av1.c
> +++ b/libavcodec/cbs_av1.c
> @@ -1058,15 +1058,33 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  AV1RawTileData *td;
>  size_t header_size;
>  int err, start_pos, end_pos, data_pos;
> +CodedBitstreamAV1Context av1ctx;
> 
>  // OBUs in the normal bitstream format must contain a size field
>  // in every OBU (in annex B it is optional, but we don't support
>  // writing that).
>  obu->header.obu_has_size_field = 1;
> +av1ctx = *priv;
> +
> +if (priv->sequence_header_ref) {
> +av1ctx.sequence_header_ref = av_buffer_ref(priv-
> >sequence_header_ref);
> +if (!av1ctx.sequence_header_ref) {
> +err = AVERROR(ENOMEM);
> +goto error;
> +}
> +}
> +
> +if (priv->frame_header_ref) {
> +av1ctx.frame_header_ref = av_buffer_ref(priv->frame_header_ref);
> +if (!av1ctx.frame_header_ref) {
> +err = AVERROR(ENOMEM);
> +goto error;
> +}
> +}
> 
>  err = cbs_av1_write_obu_header(ctx, pbc, &obu->header);
>  if (err < 0)
> -return err;
> +goto error;
> 
>  if (obu->header.obu_has_size_field) {
>  pbc_tmp = *pbc;
> @@ -1084,18 +1102,21 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  err = cbs_av1_write_sequence_header_obu(ctx, pbc,
>  
> &obu->obu.sequence_header);
>  if (err < 0)
> -return err;
> +goto error;
> 
>  av_buffer_unref(&priv->sequence_header_ref);
>  priv->sequence_header = NULL;
> 
>  err = ff_cbs_make_unit_refcounted(ctx, unit);
>  if (err < 0)
> -return err;
> +goto error;
> 
>  priv->sequence_header_ref = av_buffer_ref(unit->content_ref);
> -if (!priv->sequence_header_ref)
> -return AVERROR(ENOMEM);
> +if (!priv->sequence_header_ref) {
> +err = AVERROR(ENOMEM);
> +goto error;
> +}
> +
>  priv->sequence_header = &obu->obu.sequence_header;
>  }
>  break;
> @@ -1103,7 +1124,7 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  {
>  err = cbs_av1_write_temporal_delimiter_obu(ctx, pbc);
>  if (err < 0)
> -return err;
> +goto error;
>  }
>  break;
>  case AV1_OBU_FRAME_HEADER:
> @@ -1115,7 +1136,7 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>   
> AV1_OBU_REDUNDANT_FRAME_HEADER,
>   NULL);
>  if (err < 0)
> -return err;
> +goto error;
>  }
>  break;
>  case AV1_OBU_TILE_GROUP:
> @@ -1123,7 +1144,7 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  err = cbs_av1_write_tile_group_obu(ctx, pbc,
> &obu->obu.tile_group);
>  if (err < 0)
> -return err;
> +goto error;
> 
>  td = &obu->obu.tile_group.tile_data;
>  }
> @@ -1132,7 +1153,7 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  {
>  err = cbs_av1_write_frame_obu(ctx, pbc, &obu->obu.frame, NULL);
>  if (err < 0)
> -return err;
> +goto error;
> 
>  td = &obu->obu.frame.tile_group.tile_data;
>  }
> @@ -1141,7 +1162,7 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  {
>  err = cbs_av1_write_tile_list_obu(ctx, pbc, &obu->obu.tile_list);
>  if (err < 0)
> -return err;
> +goto error;
> 
>  td = &obu->obu.tile_list.tile_data;
>  }
> 

Re: [FFmpeg-devel] [PATCH 1/2] libavcodec/qsvenc: Let runtime to set default parameter.

[Chen, Wenbin]

> Wenbin Chen:
> > Unset qsv_h264 and qsv_hevc's default settings. Let runtime to decide
> > these parameters, so that it can choose the best parameter and ffmpeg-qsv
> > can keep up with runtime's update.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/qsvenc_h264.c | 4 ++--
> >  libavcodec/qsvenc_hevc.c | 2 +-
> >  2 files changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c
> > index 85826ae4be..fe777a6ae3 100644
> > --- a/libavcodec/qsvenc_h264.c
> > +++ b/libavcodec/qsvenc_h264.c
> > @@ -177,8 +177,8 @@ static const FFCodecDefault qsv_enc_defaults[] = {
> >  { "b", "1M"},
> >  { "refs",  "0" },
> >  // same as the x264 default
> > -{ "g", "250"   },
> > -{ "bf","3" },
> > +{ "g", "-1"},
> > +{ "bf","-1"},
> >  { "qmin",  "-1"},
> >  { "qmax",  "-1"},
> >  { "trellis",   "-1"},
> > diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c
> > index 6ec6230999..0c5bec231d 100644
> > --- a/libavcodec/qsvenc_hevc.c
> > +++ b/libavcodec/qsvenc_hevc.c
> > @@ -290,7 +290,7 @@ static const FFCodecDefault qsv_enc_defaults[] = {
> >  { "b", "1M"},
> >  { "refs",  "0" },
> >  // same as the x264 default
> 
> These comments are now wrong.

Forget these comments. I will update them in new patchset.

Thanks
Wenbin

> 
> > -{ "g", "248"   },
> > +{ "g", "-1"},
> >  { "bf","-1"},
> >  { "qmin",  "-1"},
> >  { "qmax",  "-1"},
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v4] libavcodec/cbs_av1: Add size check before parse obu

[Chen, Wenbin]

> On Wed, 2022-09-21 at 17:41 +0800, Wenbin Chen wrote:
> > cbs_av1_write_obu() check pbc size after parsing obu frame, and return
> > AVERROR(ENOSPC) if pbc is small. pbc will be reallocated and this obu
> > frame will be parsed again, but this may cause error because
> > CodedBitstreamAV1Context has already been updated, for example
> > ref_order_hint is updated and will not match the same obu frame. Now
> size
> > check is added before parsing obu frame to avoid this error.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/cbs_av1.c | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
> > index 154d9156cf..9c51a8c7c8 100644
> > --- a/libavcodec/cbs_av1.c
> > +++ b/libavcodec/cbs_av1.c
> > @@ -1075,6 +1075,9 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
> >  put_bits32(pbc, 0);
> >  }
> >
> > +if (8 * (unit->data_size + obu->obu_size) > put_bits_left(pbc))
> > +return AVERROR(ENOSPC);
> > +
> >  td = NULL;
> >  start_pos = put_bits_count(pbc);
> >
> 
> According to the comment in
> https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220329082921.7561
> 74-1-wenbin.c...@intel.com/
> , the error might still occur. I sent out
> https://ffmpeg.org/pipermail/ffmpeg-devel/2022-September/302127.html to
> restore
> CodedBitstreamAV1Context before returning AVERROR(ENOSPC). Could you
> have a look
> ?
> 
> Thanks
> Haihao

Yes, I will. Thank you for helping me on this issue.

> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: fixy typo for min/max qp reset

[Chen, Wenbin]

> Fixes: 005c7a4 ("libavcodec/qsvenc: Add max/min qp reset support in
> qsvenc")
> CC: Wenbin Chen 
> Signed-off-by: Dmitry Rogozhkin 
> ---
>  libavcodec/qsvenc.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> index 84c6e29..8bd9272 100644
> --- a/libavcodec/qsvenc.c
> +++ b/libavcodec/qsvenc.c
> @@ -1760,8 +1760,8 @@ static int update_min_max_qp(AVCodecContext
> *avctx, QSVEncContext *q)
>  if (avctx->codec_id != AV_CODEC_ID_H264)
>  return 0;
> 
> -UPDATE_PARAM(q->old_qmax, avctx->qmin);
> -UPDATE_PARAM(q->old_qmax, avctx->qmin);
> +UPDATE_PARAM(q->old_qmin, avctx->qmin);
> +UPDATE_PARAM(q->old_qmax, avctx->qmax);
>  UPDATE_PARAM(q->old_min_qp_i, q->min_qp_i);
>  UPDATE_PARAM(q->old_max_qp_i, q->max_qp_i);
>  UPDATE_PARAM(q->old_min_qp_p, q->min_qp_p);
> --
> 1.8.3.1

Thanks for your correction.
LGTM

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavcodec/cbs_av1: Add size check before parse obu

[Chen, Wenbin]

> > cbs_av1_write_unit() check pbc size after parsing obu frame, and return
> 
> It is cbs_av1_write_obu(), not cbs_av1_write_unit(), right ?
> 
> Thanks
> Haihao
> 

Sorry, it is typo. cbs_av1_write_obu is assigned to write_unit function pointer.
I will fix it and resubmit patch.

Thanks
Wenbin

> 
> > AVERROR(ENOSPC) if pbc is small. pbc will be reallocated and this obu
> > frame will be parsed again, but this may cause error because
> > CodedBitstreamAV1Context has already been updated, for example
> > ref_order_hint is updated and will not match the same obu frame. Now
> size
> > check is added before parsing obu frame to avoid this error.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/cbs_av1.c | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
> > index 154d9156cf..9c51a8c7c8 100644
> > --- a/libavcodec/cbs_av1.c
> > +++ b/libavcodec/cbs_av1.c
> > @@ -1075,6 +1075,9 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
> >  put_bits32(pbc, 0);
> >  }
> >
> > +if (8 * (unit->data_size + obu->obu_size) > put_bits_left(pbc))
> > +return AVERROR(ENOSPC);
> > +
> >  td = NULL;
> >  start_pos = put_bits_count(pbc);
> >
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 2/6] libavcodec/qsvenc: Add gop_size reset support to qsvenc

[Chen, Wenbin]

> On Thu, 2022-08-18 at 14:59 +0800, Wenbin Chen wrote:
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi   |  3 +++
> >  libavcodec/qsvenc.c | 18 +-
> >  libavcodec/qsvenc.h |  2 ++
> >  3 files changed, 22 insertions(+), 1 deletion(-)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 69fa46f3ea..9fb63856b1 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -3348,6 +3348,9 @@ Change these value to reset qsv codec's qp
> > configuration.
> >  @item @var{max_frame_size}
> >  Supported in h264_qsv and hevc_qsv.
> >  Change this value to reset qsv codec's MaxFrameSize configuration.
> > +
> > +@item @var{gop_size}
> > +Change this value to reset qsv codec's gop configuration.
> >  @end table
> >
> >  @subsection H264 options
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index c911b81e7d..b3820e4fe0 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -635,6 +635,7 @@ static int init_video_param(AVCodecContext *avctx,
> > QSVEncContext *q)
> >  q->param.mfx.CodecProfile   = q->profile;
> >  q->param.mfx.TargetUsage= avctx->compression_level;
> >  q->param.mfx.GopPicSize = FFMAX(0, avctx->gop_size);
> > +q->old_gop_size = avctx->gop_size;
> >  q->param.mfx.GopRefDist = FFMAX(-1, avctx->max_b_frames) + 1;
> >  q->param.mfx.GopOptFlag = avctx->flags &
> AV_CODEC_FLAG_CLOSED_GOP
> > ?
> >MFX_GOP_CLOSED : MFX_GOP_STRICT;
> > @@ -1692,16 +1693,31 @@ static int
> update_max_frame_size(AVCodecContext
> > *avctx, QSVEncContext *q)
> >  return updated;
> >  }
> >
> > +static int update_gop_size(AVCodecContext *avctx, QSVEncContext *q)
> > +{
> > +int updated = 0;
> > +UPDATE_PARAM(q->old_gop_size, avctx->gop_size);
> > +if (!updated)
> > +return 0;
> > +
> > +q->param.mfx.GopPicSize = FFMAX(0, avctx->gop_size);
> > +av_log(avctx, AV_LOG_DEBUG, "reset GopPicSize to %d\n",
> > +   q->param.mfx.GopPicSize);
> > +
> > +return updated;
> > +}
> > +
> >  static int update_parameters(AVCodecContext *avctx, QSVEncContext *q,
> >   const AVFrame *frame)
> >  {
> >  int needReset = 0, ret = 0;
> >
> > -if (!frame)
> > +if (!frame || avctx->codec_id == AV_CODEC_ID_MJPEG)
> >  return 0;
> 
> Better to fix mjpeg_qsv in a separate patch if mjpeg_qsv is not able to
> resetany parameter.
> 
> Thanks
> Haihao

Ok, I will update it in new patchset.

> 
> >
> >  needReset = update_qp(avctx, q);
> >  needReset |= update_max_frame_size(avctx, q);
> > +needReset |= update_gop_size(avctx, q);
> >  if (!needReset)
> >  return 0;
> >
> > diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h
> > index c452c5b806..fdedae28dd 100644
> > --- a/libavcodec/qsvenc.h
> > +++ b/libavcodec/qsvenc.h
> > @@ -237,6 +237,8 @@ typedef struct QSVEncContext {
> >  float old_b_quant_offset;
> >  // This is used for max_frame_size reset
> >  int old_max_frame_size;
> > +// This is used for gop reset
> > +int old_gop_size;
> >  } QSVEncContext;
> >
> >  int ff_qsv_enc_init(AVCodecContext *avctx, QSVEncContext *q);
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavcodec/cbs_av1: Add size check before parse obu

[Chen, Wenbin]

> > cbs_av1_write_unit() check pbc size after parsing obu frame, and return
> > AVERROR(ENOSPC) if pbc is small. pbc will be reallocated and this obu
> > frame will be parsed again, but this may cause error because
> > CodedBitstreamAV1Context has already been updated, for example
> > ref_order_hint is updated and will not match the same obu frame. Now
> size
> > check is added before parsing obu frame to avoid this error.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/cbs_av1.c | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
> > index 154d9156cf..9c51a8c7c8 100644
> > --- a/libavcodec/cbs_av1.c
> > +++ b/libavcodec/cbs_av1.c
> > @@ -1075,6 +1075,9 @@ static int
> > cbs_av1_write_obu(CodedBitstreamContext *ctx,
> >  put_bits32(pbc, 0);
> >  }
> >
> > +if (8 * (unit->data_size + obu->obu_size) > put_bits_left(pbc))
> > +return AVERROR(ENOSPC);
> > +
> >  td = NULL;
> >  start_pos = put_bits_count(pbc);
> >
> > --
> > 2.32.0
> 
> Ping

Ping 

> 
> >
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavcodec/cbs_av1: Add size check before parse obu

[Chen, Wenbin]

> cbs_av1_write_unit() check pbc size after parsing obu frame, and return
> AVERROR(ENOSPC) if pbc is small. pbc will be reallocated and this obu
> frame will be parsed again, but this may cause error because
> CodedBitstreamAV1Context has already been updated, for example
> ref_order_hint is updated and will not match the same obu frame. Now size
> check is added before parsing obu frame to avoid this error.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/cbs_av1.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
> index 154d9156cf..9c51a8c7c8 100644
> --- a/libavcodec/cbs_av1.c
> +++ b/libavcodec/cbs_av1.c
> @@ -1075,6 +1075,9 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
>  put_bits32(pbc, 0);
>  }
> 
> +if (8 * (unit->data_size + obu->obu_size) > put_bits_left(pbc))
> +return AVERROR(ENOSPC);
> +
>  td = NULL;
>  start_pos = put_bits_count(pbc);
> 
> --
> 2.32.0

Ping

> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Use parameter from AVCodecContext to reset qsv codec

[Chen, Wenbin]

> Using parameter from AVCodecContext to reset qsv codec is more suitable
> for MFXVideoENCODE_Reset()'s usage. Per-frame metadata is more suitable
> for the usage of mfxEncodeCtrl being passed to
> MFXVideoENCODE_EncodeFrameAsync(). Now change it to use the value
> from AVCodecContext.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  doc/encoders.texi   |  5 ++---
>  libavcodec/qsvenc.c | 26 --
>  2 files changed, 10 insertions(+), 21 deletions(-)
> 
> diff --git a/doc/encoders.texi b/doc/encoders.texi
> index 02a91ffe96..bf5fafd3fe 100644
> --- a/doc/encoders.texi
> +++ b/doc/encoders.texi
> @@ -3337,10 +3337,9 @@ For encoders set this flag to ON to reduce power
> consumption and GPU usage.
>  Following options can be used durning qsv encoding.
> 
>  @table @option
> -@item @var{qsv_config_qp}
> +@item @var{global_quality}
>  Supported in h264_qsv and hevc_qsv.
> -This option can be set in per-frame metadata. QP parameter can be
> dynamically
> -changed when encoding in CQP mode.
> +Change this value to reset qsv codec's qp configuration.
>  @end table
> 
>  @subsection H264 options
> diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> index 2382c2f5f7..82cb5a4865 100644
> --- a/libavcodec/qsvenc.c
> +++ b/libavcodec/qsvenc.c
> @@ -1625,34 +1625,24 @@ static int update_qp(AVCodecContext *avctx,
> QSVEncContext *q,
>   const AVFrame *frame)

"const AVFrame *frame" is not used. It can be removed.

>  {
>  int updated = 0, qp = 0, new_qp;
> -char *tail;
> -AVDictionaryEntry *entry = NULL;
> 
>  if (avctx->codec_id != AV_CODEC_ID_H264 && avctx->codec_id !=
> AV_CODEC_ID_HEVC)
>  return 0;
> 
> -entry = av_dict_get(frame->metadata, "qsv_config_qp", NULL, 0);
> -if (entry && q->param.mfx.RateControlMethod ==
> MFX_RATECONTROL_CQP) {
> -qp = strtol(entry->value, &tail, 10);
> -if (*tail) {
> -av_log(avctx, AV_LOG_WARNING, "Invalid qsv_config_qp string.
> Ignore this metadata\n");
> -return 0;
> -}
> -if (qp < 0 || qp > 51) {
> -av_log(avctx, AV_LOG_WARNING, "Invalid qp, clip to 0 ~ 51\n");
> -qp = av_clip(qp, 0, 51);
> -}
> -av_log(avctx, AV_LOG_DEBUG, "Configure qp: %d\n",qp);
> -UPDATE_PARAM(q->param.mfx.QPP, qp);
> +if (q->param.mfx.RateControlMethod == MFX_RATECONTROL_CQP) {
> +qp = avctx->global_quality / FF_QP2LAMBDA;
> +new_qp = av_clip(qp, 0, 51);
> +UPDATE_PARAM(q->param.mfx.QPP, new_qp);
>  new_qp = av_clip(qp * fabs(avctx->i_quant_factor) +
>  avctx->i_quant_offset, 0, 51);
>  UPDATE_PARAM(q->param.mfx.QPI, new_qp);
>  new_qp = av_clip(qp * fabs(avctx->b_quant_factor) +
>  avctx->b_quant_offset, 0, 51);
>  UPDATE_PARAM(q->param.mfx.QPB, new_qp);

"q->param" is passed to both "in" and "out" parameters in 
MFXVideoENCODE_Query() call,
so the value in q->param may be changed. There is a potential issue that codec 
will reset on
every frame.

I will fix these in my patch v2.

> -av_log(avctx, AV_LOG_DEBUG,
> -"using fixed qp = %d/%d/%d for idr/p/b frames\n",
> -q->param.mfx.QPI, q->param.mfx.QPP, q->param.mfx.QPB);
> +if (updated)
> +av_log(avctx, AV_LOG_DEBUG,
> +   "using fixed qp = %d/%d/%d for idr/p/b frames\n",
> +   q->param.mfx.QPI, q->param.mfx.QPP, q->param.mfx.QPB);
>  }
>  return updated;
>  }
> --
> 2.32.0
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Fix a log format issue

[Chen, Wenbin]

> > On Jul 7, 2022, at 1:57 PM, Wenbin Chen  intel@ffmpeg.org> wrote:
> >
> > Add a line feed to fix a log format issue.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> > libavcodec/qsvenc.c | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index 2382c2f5f7..81d93235d7 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -308,6 +308,7 @@ static void dump_video_param(AVCodecContext
> *avctx, QSVEncContext *q,
> > case MFX_B_REF_PYRAMID: av_log(avctx, AV_LOG_VERBOSE,
> "pyramid");   break;
> > default:av_log(avctx, AV_LOG_VERBOSE, "auto");  
> > break;
> > }
> > +av_log(avctx, AV_LOG_VERBOSE, "\n");
> 
> It’s not thread safe. There are multiple such cases, better be fixed together.

Ok, I will fix them together.

Thanks

> 
> >
> > av_log(avctx, AV_LOG_VERBOSE,
> >"MinQPI: %"PRIu8"; MaxQPI: %"PRIu8"; MinQPP: %"PRIu8";
> MaxQPP: %"PRIu8"; MinQPB: %"PRIu8"; MaxQPB: %"PRIu8"\n",
> > --
> > 2.32.0
> >
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavcodec/qsvenc: Enable fixed QP configure in qsv CQP runtime

[Chen, Wenbin]

> Quoting Chen, Wenbin (2022-07-04 08:33:49)
> > > Why is this using frame metadata rather than the AVVideoEncParams
> side
> > > data?
> >
> > The usage of AVVideoEncParams relates to the "qp" variable in
> mfxEncodeCtrl which is passed
> > to MFXVideoENCODE_encoderFrameAsync(). This variable in qsv is for per-
> frame QP
> > configuration.
> > There are other parameter changing supports I want to add besides QP, for
> > example, gop_size, max_frame_size, intra_refresh. These parameter
> configurations are not
> > all included in mfxEncodeCtrl, so I choose to use MFXVideoENCODE_Reset()
> to do this. This
> > code changes the encoding parameters which means these changes are
> applied to all
> > the following frames, but AVVideoEncParams is per-frame configuration,
> so I think
> > AVVideoEncParams is not suitable for this.
> 
> AVFrame metadata is also per-frame, so your logic does not make sense to
> me.
> 
> You could also just update the AVCodecContext/private context values
> directly or using AVOptions.

This is a possible way. I will change it. Thanks for your advice.

> 
> --
> Anton Khirnov
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavcodec/qsvenc: Enable fixed QP configure in qsv CQP runtime

[Chen, Wenbin]

> Quoting Wenbin Chen (2022-06-23 07:32:42)
> > From: Yue Heng 
> >
> > Enable dynamic QP configuration in runtime on qsv encoder. Through
> > AVFrame->metadata, we can set key "qsv_config_qp" to change QP
> > configuration when we encode video in CQP mode.
> >
> > Signed-off-by: Yue Heng 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi   | 10 +
> >  libavcodec/qsvenc.c | 89
> +
> >  2 files changed, 99 insertions(+)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 1850c99fe9..02a91ffe96 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -,6 +,16 @@ Forcing I frames as IDR frames.
> >  For encoders set this flag to ON to reduce power consumption and GPU
> usage.
> >  @end table
> >
> > +@subsection Runtime Options
> > +Following options can be used durning qsv encoding.
> > +
> > +@table @option
> > +@item @var{qsv_config_qp}
> > +Supported in h264_qsv and hevc_qsv.
> > +This option can be set in per-frame metadata. QP parameter can be
> dynamically
> > +changed when encoding in CQP mode.
> > +@end table
> > +
> >  @subsection H264 options
> >  These options are used by h264_qsv
> >
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index 902bada55b..2382c2f5f7 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -146,6 +146,14 @@ static const struct {
> >  { MFX_RATECONTROL_QVBR,"QVBR" },
> >  };
> >
> > +#define UPDATE_PARAM(a, b)  \
> > +do {\
> > +if ((a) != (b)) {   \
> > +a = b;  \
> > +updated = 1;\
> > +}   \
> > +} while (0) \
> > +
> >  static const char *print_ratecontrol(mfxU16 rc_mode)
> >  {
> >  int i;
> > @@ -1613,6 +1621,83 @@ static int set_roi_encode_ctrl(AVCodecContext
> *avctx, const AVFrame *frame,
> >  return 0;
> >  }
> >
> > +static int update_qp(AVCodecContext *avctx, QSVEncContext *q,
> > + const AVFrame *frame)
> > +{
> > +int updated = 0, qp = 0, new_qp;
> > +char *tail;
> > +AVDictionaryEntry *entry = NULL;
> > +
> > +if (avctx->codec_id != AV_CODEC_ID_H264 && avctx->codec_id !=
> AV_CODEC_ID_HEVC)
> > +return 0;
> > +
> > +entry = av_dict_get(frame->metadata, "qsv_config_qp", NULL, 0);
> 
> Why is this using frame metadata rather than the AVVideoEncParams side
> data?

The usage of AVVideoEncParams relates to the "qp" variable in mfxEncodeCtrl 
which is passed
to MFXVideoENCODE_encoderFrameAsync(). This variable in qsv is for per-frame QP
configuration.
There are other parameter changing supports I want to add besides QP, for
example, gop_size, max_frame_size, intra_refresh. These parameter 
configurations are not
all included in mfxEncodeCtrl, so I choose to use MFXVideoENCODE_Reset() to do 
this. This
code changes the encoding parameters which means these changes are applied to 
all
the following frames, but AVVideoEncParams is per-frame configuration, so I 
think
AVVideoEncParams is not suitable for this.

> 
> Frame metadata should not be used in encoders at all. Longer term it
> should not be used at all by anything.
> 
> --
> Anton Khirnov
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/mpegvideo_enc: Fix a chroma mb size error in sse_mb()

[Chen, Wenbin]

> On Fri, Jul 01, 2022 at 01:34:34PM +0800, Wenbin Chen wrote:
> > For 422 frames we should not use hard coded 8 to calculate mb size for
> > uv plane. Chroma shift should be taken into consideration to be
> > compatiple with different sampling format.
> >
> > The error is reported by fate test when av_cpu_max_align() return 64
> > on the platform supporting AVX512. This is a hidden error and it is
> > exposed after commit 17a59a634c39b00a680c6ebbaea58db95594d13d.
> >
> > mpeg2enc has a mechanism to reuse frames. When it computes SSE (sum
> of
> > squared error) on current mb, reconstructed mb will be wrote to the
> > previous mb space, so that the memory can be saved. However if the align
> > is 64, the frame is shared in somewhere else, so the frame cannot be
> > reused and a new frame to store reconstrued data is created. Because the
> > height of mb is wrong when compute sse on 422 frame, starting from the
> > second line of macro block, changed data is read when frame is reused
> > (we need to read row 16 rather than row 8 if frame is 422), and unchanged
> > data is read when frame is not reused (a new frame is created so the
> > original frame will not be changed).
> >
> > That is why commit 17a59a634c39b00a680c6ebbaea58db95594d13d
> exposes this
> > issue, because it add av_cpu_max_align() and this function return 64 on
> > platform supporting AVX512 which lead to creating a frame in mpeg2enc,
> > and this lead to the different outputs.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/mpegvideo_enc.c | 29 +--
> >  tests/ref/seek/vsynth_lena-mpeg2-422   | 40 +-
> >  tests/ref/vsynth/vsynth1-mpeg2-422 |  8 +++---
> >  tests/ref/vsynth/vsynth2-mpeg2-422 |  8 +++---
> >  tests/ref/vsynth/vsynth3-mpeg2-422 |  8 +++---
> >  tests/ref/vsynth/vsynth_lena-mpeg2-422 |  8 +++---
> >  6 files changed, 56 insertions(+), 45 deletions(-)
> >
> > diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
> > index d6a85a037a..c9d9e2a764 100644
> > --- a/libavcodec/mpegvideo_enc.c
> > +++ b/libavcodec/mpegvideo_enc.c
> > @@ -2558,24 +2558,35 @@ static int sse(MpegEncContext *s, uint8_t *src1,
> uint8_t *src2, int w, int h, in
> >  static int sse_mb(MpegEncContext *s){
> >  int w= 16;
> >  int h= 16;
> > +int chroma_mb_w = w >> s->chroma_x_shift;
> > +int chroma_mb_h = h >> s->chroma_y_shift;
> >
> >  if(s->mb_x*16 + 16 > s->width ) w= s->width - s->mb_x*16;
> >  if(s->mb_y*16 + 16 > s->height) h= s->height- s->mb_y*16;
> >
> >  if(w==16 && h==16)
> >if(s->avctx->mb_cmp == FF_CMP_NSSE){
> 
> > -return s->mecc.nsse[0](s, s->new_picture->data[0] + s->mb_x * 16 + 
> > s-
> >mb_y * s->linesize   * 16, s->dest[0], s->linesize,   16) +
> > +return s->mecc.nsse[0](s, s->new_picture->data[0] + s->mb_x * 16 +
> s->mb_y * s->linesize * 16,
> > +   s->dest[0], s->linesize, 16) +
> 
> This doesnt belong in this patch, its not a functional change and makes it
> harder to see what is changed
> 
> please seperate this in a seperate patch if you want to suggest this
> whitespace change
> 
> thx

Ok, I will separate it into two patches and resubmit them.

Thanks

> 
> 
> > -   s->mecc.nsse[1](s, s->new_picture->data[1] + s->mb_x *  8 + 
> > s-
> >mb_y * s->uvlinesize *  8, s->dest[1], s->uvlinesize,  8) +
> > -   s->mecc.nsse[1](s, s->new_picture->data[2] + s->mb_x *  8 + 
> > s-
> >mb_y * s->uvlinesize *  8, s->dest[2], s->uvlinesize,  8);
> > +   s->mecc.nsse[1](s, s->new_picture->data[1] + s->mb_x *
> chroma_mb_w + s->mb_y * s->uvlinesize * chroma_mb_h,
> > +   s->dest[1], s->uvlinesize, chroma_mb_h) +
> > +   s->mecc.nsse[1](s, s->new_picture->data[2] + s->mb_x *
> chroma_mb_w + s->mb_y * s->uvlinesize * chroma_mb_h,
> > +   s->dest[2], s->uvlinesize, chroma_mb_h);
> 
> [...]
> --
> Michael GnuPG fingerprint:
> 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> If you fake or manipulate statistics in a paper in physics you will never
> get a job again.
> If you fake or manipulate statistics in a paper in medicin you will get
> a job for life at the pharma industry.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/mpegvideo_enc: Fix a chroma mb size error in sse_mb()

[Chen, Wenbin]

> On Fri, Jul 1, 2022 at 10:00 AM Xiang, Haihao <
> haihao.xiang-at-intel@ffmpeg.org> wrote:
> 
> > On Fri, 2022-07-01 at 13:34 +0800, Wenbin Chen wrote:
> > > For 422 frames we should not use hard coded 8 to calculate mb size for
> > > uv plane. Chroma shift should be taken into consideration to be
> > > compatiple with different sampling format.
> > >
> > > The error is reported by fate test when av_cpu_max_align() return 64
> > > on the platform supporting AVX512. This is a hidden error and it is
> > > exposed after commit 17a59a634c39b00a680c6ebbaea58db95594d13d.
> > >
> > > mpeg2enc has a mechanism to reuse frames. When it computes SSE (sum
> of
> > > squared error) on current mb, reconstructed mb will be wrote to the
> > > previous mb space, so that the memory can be saved. However if the
> align
> > > is 64, the frame is shared in somewhere else, so the frame cannot be
> > > reused and a new frame to store reconstrued data is created. Because
> the
> > > height of mb is wrong when compute sse on 422 frame, starting from the
> > > second line of macro block, changed data is read when frame is reused
> > > (we need to read row 16 rather than row 8 if frame is 422), and
> unchanged
> > > data is read when frame is not reused (a new frame is created so the
> > > original frame will not be changed).
> > >
> > > That is why commit 17a59a634c39b00a680c6ebbaea58db95594d13d
> exposes this
> > > issue, because it add av_cpu_max_align() and this function return 64 on
> > > platform supporting AVX512 which lead to creating a frame in mpeg2enc,
> > > and this lead to the different outputs.
> > >
> > > Signed-off-by: Wenbin Chen 
> > > ---
> > >  libavcodec/mpegvideo_enc.c | 29 +--
> > >  tests/ref/seek/vsynth_lena-mpeg2-422   | 40 +-
> > >  tests/ref/vsynth/vsynth1-mpeg2-422 |  8 +++---
> > >  tests/ref/vsynth/vsynth2-mpeg2-422 |  8 +++---
> > >  tests/ref/vsynth/vsynth3-mpeg2-422 |  8 +++---
> > >  tests/ref/vsynth/vsynth_lena-mpeg2-422 |  8 +++---
> > >  6 files changed, 56 insertions(+), 45 deletions(-)
> > >
> > > diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
> > > index d6a85a037a..c9d9e2a764 100644
> > > --- a/libavcodec/mpegvideo_enc.c
> > > +++ b/libavcodec/mpegvideo_enc.c
> > > @@ -2558,24 +2558,35 @@ static int sse(MpegEncContext *s, uint8_t
> *src1,
> > > uint8_t *src2, int w, int h, in
> > >  static int sse_mb(MpegEncContext *s){
> > >  int w= 16;
> > >  int h= 16;
> > > +int chroma_mb_w = w >> s->chroma_x_shift;
> > > +int chroma_mb_h = h >> s->chroma_y_shift;
> > >
> > >  if(s->mb_x*16 + 16 > s->width ) w= s->width - s->mb_x*16;
> > >  if(s->mb_y*16 + 16 > s->height) h= s->height- s->mb_y*16;
> > >
> > >  if(w==16 && h==16)
> > >if(s->avctx->mb_cmp == FF_CMP_NSSE){
> > > -return s->mecc.nsse[0](s, s->new_picture->data[0] + s->mb_x *
> > 16 + s-
> > > >mb_y * s->linesize   * 16, s->dest[0], s->linesize,   16) +
> > > -   s->mecc.nsse[1](s, s->new_picture->data[1] + s->mb_x *
> > 8 + s-
> > > >mb_y * s->uvlinesize *  8, s->dest[1], s->uvlinesize,  8) +
> > > -   s->mecc.nsse[1](s, s->new_picture->data[2] + s->mb_x *
> > 8 + s-
> > > >mb_y * s->uvlinesize *  8, s->dest[2], s->uvlinesize,  8);
> > > +return s->mecc.nsse[0](s, s->new_picture->data[0] + s->mb_x *
> > 16 + s-
> > > >mb_y * s->linesize * 16,
> > > +   s->dest[0], s->linesize, 16) +
> > > +   s->mecc.nsse[1](s, s->new_picture->data[1] + s->mb_x *
> > > chroma_mb_w + s->mb_y * s->uvlinesize * chroma_mb_h,
> > > +   s->dest[1], s->uvlinesize, chroma_mb_h) +
> > > +   s->mecc.nsse[1](s, s->new_picture->data[2] + s->mb_x *
> > > chroma_mb_w + s->mb_y * s->uvlinesize * chroma_mb_h,
> > > +   s->dest[2], s->uvlinesize, chroma_mb_h);
> > >}else{
> > > -return s->mecc.sse[0](NULL, s->new_picture->data[0] + s->mb_x *
> > 16 +
> > > s->mb_y * s->linesize   * 16, s->dest[0], s->linesize,   16) +
> > > -   s->mecc.sse[1](NULL, s->new_picture->data[1] + s->mb_x
> > *  8 +
> > > s->mb_y * s->uvlinesize *  8, s->dest[1], s->uvlinesize,  8) +
> > > -   s->mecc.sse[1](NULL, s->new_picture->data[2] + s->mb_x
> > *  8 +
> > > s->mb_y * s->uvlinesize *  8, s->dest[2], s->uvlinesize,  8);
> > > +return s->mecc.sse[0](NULL, s->new_picture->data[0] + s->mb_x *
> > 16 +
> > > s->mb_y * s->linesize * 16,
> > > +  s->dest[0], s->linesize, 16) +
> > > +   s->mecc.sse[1](NULL, s->new_picture->data[1] + s->mb_x *
> > > chroma_mb_w + s->mb_y * s->uvlinesize * chroma_mb_h,
> > > +  s->dest[1], s->uvlinesize, chroma_mb_h) +
> > > +   s->mecc.sse[1](NULL, s->new_picture->data[2] + s->mb_x *
> > > chroma_mb_w + s->mb_y * s->uvlinesize * chroma_mb_h,
> > > +  

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Enable fixed QP configure in qsv CQP runtime

[Chen, Wenbin]

> On Wed, 2022-06-15 at 15:25 +0800, Wenbin Chen wrote:
> > From: Yue Heng 
> >
> > Enable dynamic QP configuration in runtime on qsv encoder. Through
> > AVFrame->metadata, we can set key "qsv_config_qp" to change QP
> > configuration when we encode video in CQP mode.
> >
> > Signed-off-by: Yue Heng 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi   |  9 +++
> >  libavcodec/qsvenc.c | 59
> +
> >  2 files changed, 68 insertions(+)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 9796a606fa..1eb75e199a 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -,6 +,15 @@ Forcing I frames as IDR frames.
> >  For encoders set this flag to ON to reduce power consumption and GPU
> usage.
> >  @end table
> >
> > +@subsection Runtime Options
> > +Following options can be used durning qsv encoding.
> > +
> > +@table @option
> > +@item @var{qsv_config_qp}
> > +This option can be set in per-frame metadata. QP parameter can be
> dynamically
> > +changed when encode in CQP mode.
> > +@end table
> > +
> >  @subsection H264 options
> >  These options are used by h264_qsv
> >
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index 2b3b06767d..52462950be 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -146,6 +146,14 @@ static const struct {
> >  { MFX_RATECONTROL_QVBR,"QVBR" },
> >  };
> >
> > +#define UPDATE_PARAM(a, b)  \
> > +do {\
> > +if ((a) != (b)) {   \
> > +a = b;  \
> > +updated = 1;\
> > +}   \
> > +} while (0) \
> > +
> >  static const char *print_ratecontrol(mfxU16 rc_mode)
> >  {
> >  int i;
> > @@ -1520,6 +1528,53 @@ static void
> print_interlace_msg(AVCodecContext *avctx,
> > QSVEncContext *q)
> >  }
> >  }
> >
> > +static int update_qp(AVCodecContext *avctx, QSVEncContext *q,
> > + const AVFrame *frame)
> > +{
> > +int updated = 0, qp = 0, new_qp;
> > +AVDictionaryEntry *entry = av_dict_get(frame->metadata,
> "qsv_config_qp",
> > +   NULL, 0);
> > +if (entry && q->param.mfx.RateControlMethod ==
> MFX_RATECONTROL_CQP) {
> > +qp = atoi(entry->value);
> 
> I think it is better to use strtol() because atoi() doesn't detect errors. See
> `man atoi`

Will change to strtol().

> 
> 
> > +av_log(avctx, AV_LOG_DEBUG, "Configure qp: %d\n",qp);
> > +if (qp < 0 || qp > 51)
> > +av_log(avctx, AV_LOG_WARNING, "Invalid qp, clip to 0 ~ 51\n");
> 
> Is this feature for H264 and H265 only ?

I will add a codec check here.

> 
> > +
> > +qp = av_clip(qp, 0, 51);
> > +UPDATE_PARAM(q->param.mfx.QPP, qp);
> > +new_qp = av_clip(qp * fabs(avctx->i_quant_factor) +
> > +avctx->i_quant_offset, 0, 51);
> > +UPDATE_PARAM(q->param.mfx.QPI, new_qp);
> > +new_qp = av_clip(qp * fabs(avctx->b_quant_factor) +
> > +avctx->b_quant_offset, 0, 51);
> > +UPDATE_PARAM(q->param.mfx.QPB, new_qp);
> > +av_log(avctx, AV_LOG_DEBUG,
> > +"using fixed qp = %d/%d/%d for idr/p/b frames\n",
> > +q->param.mfx.QPI, q->param.mfx.QPP, q->param.mfx.QPB);
> > +}
> > +return updated;
> > +}
> > +
> > +static int update_parameters(AVCodecContext *avctx, QSVEncContext *q,
> > + const AVFrame *frame)
> > +{
> > +int needReset = 0, ret = 0;
> > +
> > +if (!frame)
> > +return 0;
> > +
> > +needReset = update_qp(avctx, q, frame);
> > +if (needReset) {
> > +q->param.ExtParam= q->extparam_internal;
> > +q->param.NumExtParam = q->nb_extparam_internal;
> 
> User might provide external parameters in avctx->hwaccel_context. Are
> these
> parameters not required in MFXVideoENCODE_Reset() ?

Miss the user provided parameters. I will add them as well.

> 
> Thanks
> Haihao

Thanks for your review. I will fix these in new patch.

Thanks
Wenbin

> 
> > +av_log(avctx, AV_LOG_DEBUG, "Parameter change, call msdk
> reset.\n");
> > +ret = MFXVideoENCODE_Reset(q->session, &q->param);
> > +if (ret < 0)
> > +return ff_qsv_print_error(avctx, ret, "Error during 
> > resetting");
> > +}
> > +return 0;
> > +}
> > +
> >  static int encode_frame(AVCodecContext *avctx, QSVEncContext *q,
> >  const AVFrame *frame)
> >  {
> > @@ -1630,6 +1685,10 @@ int ff_qsv_encode(AVCodecContext *avctx,
> QSVEncContext
> > *q,
> >  {
> >  int ret;
> >
> > +ret = update_parameters(avctx, q, frame);
> > +if (ret < 0)
> > +return ret;
> > +
> >  ret = encode_frame(avctx, q, frame);
> >  if (ret < 0)
> >  return ret;
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http

Re: [FFmpeg-devel] [PATCH v3] libavcodec/qsvenc: add ROI support to qsv encoder

[Chen, Wenbin]

> Use The mfxEncoderCtrl parameter to enable ROI. Get side data
> "AVRegionOfInterest" and use it to configure "mfxExtEncoderROI" which is
> the MediaSDK's ROI configuration.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/qsv_internal.h |  4 ++
>  libavcodec/qsvenc.c   | 86
> +++
>  2 files changed, 90 insertions(+)
> 
> diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> index e2aecdcbd6..8131acdae9 100644
> --- a/libavcodec/qsv_internal.h
> +++ b/libavcodec/qsv_internal.h
> @@ -51,6 +51,9 @@
>  #define ASYNC_DEPTH_DEFAULT 4   // internal parallelism
> 
>  #define QSV_MAX_ENC_PAYLOAD 2   // # of mfxEncodeCtrl payloads
> supported
> +#define QSV_MAX_ENC_EXTPARAM 2
> +
> +#define QSV_MAX_ROI_NUM 256
> 
>  #define QSV_MAX_FRAME_EXT_PARAMS 4
> 
> @@ -83,6 +86,7 @@ typedef struct QSVFrame {
>  int num_ext_params;
> 
>  mfxPayload *payloads[QSV_MAX_ENC_PAYLOAD]; ///< used for
> enc_ctrl.Payload
> +mfxExtBuffer *extparam[QSV_MAX_ENC_EXTPARAM]; ///< used for
> enc_ctrl.ExtParam
> 
>  int queued;
>  int used;
> diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> index 2b3b06767d..0e88a10489 100644
> --- a/libavcodec/qsvenc.c
> +++ b/libavcodec/qsvenc.c
> @@ -1373,15 +1373,29 @@ static void
> free_encoder_ctrl_payloads(mfxEncodeCtrl* enc_ctrl)
>  }
>  }
> 
> +static void free_encoder_ctrl_extparam(mfxEncodeCtrl* enc_ctrl)
> +{
> +if (enc_ctrl) {
> +int i;
> +for (i = 0; i < enc_ctrl->NumExtParam && i <
> QSV_MAX_ENC_EXTPARAM; i++) {
> +if (enc_ctrl->ExtParam[i])
> +av_freep(&(enc_ctrl->ExtParam[i]));
> +}
> +enc_ctrl->NumExtParam = 0;
> +}
> +}
> +
>  static void clear_unused_frames(QSVEncContext *q)
>  {
>  QSVFrame *cur = q->work_frames;
>  while (cur) {
>  if (cur->used && !cur->surface.Data.Locked) {
>  free_encoder_ctrl_payloads(&cur->enc_ctrl);
> +free_encoder_ctrl_extparam(&cur->enc_ctrl);
>  //do not reuse enc_ctrl from previous frame
>  memset(&cur->enc_ctrl, 0, sizeof(cur->enc_ctrl));
>  cur->enc_ctrl.Payload = cur->payloads;
> +cur->enc_ctrl.ExtParam = cur->extparam;
>  if (cur->frame->format == AV_PIX_FMT_QSV) {
>  av_frame_unref(cur->frame);
>  }
> @@ -1419,6 +1433,7 @@ static int get_free_frame(QSVEncContext *q,
> QSVFrame **f)
>  return AVERROR(ENOMEM);
>  }
>  frame->enc_ctrl.Payload = frame->payloads;
> +frame->enc_ctrl.ExtParam = frame->extparam;
>  *last = frame;
> 
>  *f = frame;
> @@ -1520,6 +1535,68 @@ static void print_interlace_msg(AVCodecContext
> *avctx, QSVEncContext *q)
>  }
>  }
> 
> +static int set_roi_encode_ctrl(AVCodecContext *avctx, const AVFrame
> *frame,
> +   mfxEncodeCtrl *enc_ctrl)
> +{
> +AVFrameSideData *sd = NULL;
> +int mb_size;
> +
> +if (avctx->codec_id == AV_CODEC_ID_H264) {
> +mb_size = 16;
> +} else if (avctx->codec_id == AV_CODEC_ID_H265) {
> +mb_size = 32;
> +} else {
> +return 0;
> +}

Don't need {} if there is one line. I will update this patch.

> +
> +if (frame)
> +sd = av_frame_get_side_data(frame,
> AV_FRAME_DATA_REGIONS_OF_INTEREST);
> +
> +if (sd) {
> +mfxExtEncoderROI *enc_roi = NULL;
> +AVRegionOfInterest *roi;
> +uint32_t roi_size;
> +int nb_roi, i;
> +
> +roi = (AVRegionOfInterest *)sd->data;
> +roi_size = roi->self_size;
> +if (!roi_size || sd->size % roi_size) {
> +av_log(avctx, AV_LOG_ERROR, "Invalid ROI Data.\n");
> +return AVERROR(EINVAL);
> +}
> +nb_roi = sd->size / roi_size;
> +if (nb_roi > QSV_MAX_ROI_NUM) {
> +av_log(avctx, AV_LOG_WARNING, "More ROIs set than "
> +"supported by driver (%d > %d).\n",
> +nb_roi, QSV_MAX_ROI_NUM);
> +nb_roi = QSV_MAX_ROI_NUM;
> +}
> +
> +enc_roi = av_mallocz(sizeof(*enc_roi));
> +if (!enc_roi)
> +return AVERROR(ENOMEM);
> +enc_roi->Header.BufferId = MFX_EXTBUFF_ENCODER_ROI;
> +enc_roi->Header.BufferSz = sizeof(*enc_roi);
> +enc_roi->NumROI  = nb_roi;
> +enc_roi->ROIMode = MFX_ROI_MODE_QP_DELTA;
> +for (i = 0; i < nb_roi; i++) {
> +roi = (AVRegionOfInterest *)(sd->data + roi_size * i);
> +enc_roi->ROI[i].Top= FFALIGN(roi->top, mb_size);
> +enc_roi->ROI[i].Bottom = FFALIGN(roi->bottom, mb_size);
> +enc_roi->ROI[i].Left   = FFALIGN(roi->left, mb_size);
> +enc_roi->ROI[i].Right  = FFALIGN(roi->right, mb_size);
> +enc_roi->ROI[i].DeltaQP =
> +roi->qoffset.num * 51 / roi->qoffset.den;
> +av_log(avctx, AV_LOG_DEBUG, "ROI: (%d,%d)-(%d,%d) -> %+d.\n",
> +

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: add ROI support to qsv encoder

[Chen, Wenbin]

> On Fri, 2022-05-06 at 13:49 +0800, Wenbin Chen wrote:
> > Use the mfxEncoderCtrl parameter to enable ROI. Get side data
> > "AVRegionOfInterest"
> > from filter "addroi" and use it to configure "mfxExtEncoderROI" which is
> > the MediaSDK's ROI configuration.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/qsv_internal.h |  4 ++
> >  libavcodec/qsvenc.c   | 88
> +++
> >  2 files changed, 92 insertions(+)
> >
> > diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> > index e2aecdcbd6..8131acdae9 100644
> > --- a/libavcodec/qsv_internal.h
> > +++ b/libavcodec/qsv_internal.h
> > @@ -51,6 +51,9 @@
> >  #define ASYNC_DEPTH_DEFAULT 4   // internal parallelism
> >
> >  #define QSV_MAX_ENC_PAYLOAD 2   // # of mfxEncodeCtrl payloads
> supported
> > +#define QSV_MAX_ENC_EXTPARAM 2
> > +
> > +#define QSV_MAX_ROI_NUM 256
> >
> >  #define QSV_MAX_FRAME_EXT_PARAMS 4
> >
> > @@ -83,6 +86,7 @@ typedef struct QSVFrame {
> >  int num_ext_params;
> >
> >  mfxPayload *payloads[QSV_MAX_ENC_PAYLOAD]; ///< used for
> enc_ctrl.Payload
> > +mfxExtBuffer *extparam[QSV_MAX_ENC_EXTPARAM]; ///< used for
> > enc_ctrl.ExtParam
> >
> >  int queued;
> >  int used;
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index fbb22ca436..9abead604b 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -1501,15 +1501,29 @@ static void
> free_encoder_ctrl_payloads(mfxEncodeCtrl*
> > enc_ctrl)
> >  }
> >  }
> >
> > +static void free_encoder_ctrl_extparam(mfxEncodeCtrl* enc_ctrl)
> > +{
> > +if (enc_ctrl) {
> > +int i;
> > +for (i = 0; i < enc_ctrl->NumExtParam && i <
> QSV_MAX_ENC_EXTPARAM;
> > i++) {
> > +if (enc_ctrl->ExtParam[i])
> > +av_freep(&(enc_ctrl->ExtParam[i]));
> > +}
> > +enc_ctrl->NumExtParam = 0;
> > +}
> > +}
> > +
> >  static void clear_unused_frames(QSVEncContext *q)
> >  {
> >  QSVFrame *cur = q->work_frames;
> >  while (cur) {
> >  if (cur->used && !cur->surface.Data.Locked) {
> >  free_encoder_ctrl_payloads(&cur->enc_ctrl);
> > +free_encoder_ctrl_extparam(&cur->enc_ctrl);
> >  //do not reuse enc_ctrl from previous frame
> >  memset(&cur->enc_ctrl, 0, sizeof(cur->enc_ctrl));
> >  cur->enc_ctrl.Payload = cur->payloads;
> > +cur->enc_ctrl.ExtParam = cur->extparam;
> >  if (cur->frame->format == AV_PIX_FMT_QSV) {
> >  av_frame_unref(cur->frame);
> >  }
> > @@ -1547,6 +1561,7 @@ static int get_free_frame(QSVEncContext *q,
> QSVFrame
> > **f)
> >  return AVERROR(ENOMEM);
> >  }
> >  frame->enc_ctrl.Payload = frame->payloads;
> > +frame->enc_ctrl.ExtParam = frame->extparam;
> >  *last = frame;
> >
> >  *f = frame;
> > @@ -1648,6 +1663,70 @@ static void print_interlace_msg(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  }
> >  }
> >
> > +static int set_roi_encode_ctrl(AVCodecContext *avctx, const AVFrame
> *frame,
> > +   mfxEncodeCtrl *enc_ctrl)
> > +{
> > +#if QSV_VERSION_ATLEAST(1, 22)
> 
> This check can be removed now because the minimal version is 1.28
> 
> > +AVFrameSideData *sd = NULL;
> > +int mb_size;
> > +
> > +if (avctx->codec_id == AV_CODEC_ID_H264) {
> > +mb_size = 16;
> > +} else if (avctx->codec_id == AV_CODEC_ID_H265) {
> > +mb_size = 32;
> > +} else {
> > +return 0;
> > +}
> > +
> > +if (frame)
> > +sd = av_frame_get_side_data(frame,
> > AV_FRAME_DATA_REGIONS_OF_INTEREST);
> > +
> > +if (sd) {
> > +mfxExtEncoderROI *enc_roi = NULL;
> > +AVRegionOfInterest *roi;
> > +uint32_t roi_size;
> > +int nb_roi, i;
> > +
> > +roi = (AVRegionOfInterest *)sd->data;
> > +roi_size = roi->self_size;
> > +if (!roi_size || sd->size % roi_size) {
> > +av_log(avctx, AV_LOG_ERROR, "Invalid ROI Data.\n");
> > +return AVERROR(EINVAL);
> > +}
> > +nb_roi = sd->size / roi_size;
> > +if (nb_roi > QSV_MAX_ROI_NUM) {
> > +av_log(avctx, AV_LOG_WARNING, "More ROIs set than "
> > +"supported by driver (%d > %d).\n",
> > +nb_roi, QSV_MAX_ROI_NUM);
> > +nb_roi = QSV_MAX_ROI_NUM;
> > +}
> > +
> > +enc_roi = av_mallocz(sizeof(*enc_roi));
> > +if (!enc_roi)
> > +return AVERROR(ENOMEM);
> > +enc_roi->Header.BufferId = MFX_EXTBUFF_ENCODER_ROI;
> > +enc_roi->Header.BufferSz = sizeof(*enc_roi);
> > +enc_roi->NumROI  = nb_roi;
> > +enc_roi->ROIMode = MFX_ROI_MODE_QP_DELTA;
> > +for (i = 0; i < nb_roi; i++) {
> > +roi = (AVRegionOfInterest *)(sd->data + roi_size * i);
> > +enc_roi->ROI[i].Top= FFALIGN(roi->top, mb_size);
> > +enc_

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc: Add min/max QP control options for I/P/B frame

[Chen, Wenbin]

> On Fri, 2022-05-06 at 10:19 +0800, Wenbin Chen wrote:
> > From: Yue Heng 
> >
> > To do more accurate QP control, add min/max QP control on I/P/B frame
> > separately to qsv encoder. qmax and qmin still work but newly-added
> > options have higher priority.
> >
> > Signed-off-by: Yue Heng 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi   | 18 ++
> >  libavcodec/qsvenc.c | 21 +++--
> >  libavcodec/qsvenc.h | 12 
> >  3 files changed, 49 insertions(+), 2 deletions(-)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 966032a720..86bcdbc04b 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -3282,6 +3282,24 @@ Forcing I frames as IDR frames.
> >
> >  @item @var{low_power}
> >  For encoders set this flag to ON to reduce power consumption and GPU
> usage.
> > +
> > +@item @var{max_qp_i}
> > +Maximum video quantizer scale for I frame.
> > +
> > +@item @var{min_qp_i}
> > +Minimum video quantizer scale for I frame.
> > +
> > +@item @var{max_qp_p}
> > +Maximum video quantizer scale for P frame.
> > +
> > +@item @var{min_qp_p}
> > +Minimum video quantizer scale for P frame.
> > +
> > +@item @var{max_qp_b}
> > +Maximum video quantizer scale for B frame.
> > +
> > +@item @var{min_qp_b}
> > +Minimum video quantizer scale for B frame.
> >  @end table
> >
> 
> Are these common options for all codecs ? If not, could you please update
> your
> patch ?
> 
> Thanks
> Haihao

Ok, I will update this patch and submit again.

Thanks
Wenbin
> 
> >  @subsection H264 options
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index fbb22ca436..a3e9bb4583 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -930,8 +930,13 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  q->extco2.BRefType = q->b_strategy ? MFX_B_REF_PYRAMID :
> > MFX_B_REF_OFF;
> >  #endif
> >  #if QSV_VERSION_ATLEAST(1, 9)
> > -if (avctx->qmin >= 0 && avctx->qmax >= 0 && avctx->qmin > 
> > avctx-
> > >qmax) {
> > -av_log(avctx, AV_LOG_ERROR, "qmin and or qmax are set but
> > invalid, please make sure min <= max\n");
> > +if ((avctx->qmin >= 0 && avctx->qmax >= 0 && avctx->qmin > 
> > avctx-
> > >qmax) ||
> > +(q->max_qp_i >= 0 && q->min_qp_i >= 0 && q->min_qp_i > q-
> > >max_qp_i) ||
> > +(q->max_qp_p >= 0 && q->min_qp_p >= 0 && q->min_qp_p > q-
> > >max_qp_p) ||
> > +(q->max_qp_b >= 0 && q->min_qp_b >= 0 && q->min_qp_b > q-
> > >max_qp_b)) {
> > +av_log(avctx, AV_LOG_ERROR,
> > +   "qmin and or qmax are set but invalid,"
> > +   " please make sure min <= max\n");
> >  return AVERROR(EINVAL);
> >  }
> >  if (avctx->qmin >= 0) {
> > @@ -942,6 +947,18 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  q->extco2.MaxQPI = avctx->qmax > 51 ? 51 : avctx->qmax;
> >  q->extco2.MaxQPP = q->extco2.MaxQPB = q->extco2.MaxQPI;
> >  }
> > +if (q->min_qp_i >= 0)
> > +q->extco2.MinQPI = q->min_qp_i > 51 ? 51 : q->min_qp_i;
> > +if (q->max_qp_i >= 0)
> > +q->extco2.MaxQPI = q->max_qp_i > 51 ? 51 : q->max_qp_i;
> > +if (q->min_qp_p >= 0)
> > +q->extco2.MinQPP = q->min_qp_p > 51 ? 51 : q->min_qp_p;
> > +if (q->max_qp_p >= 0)
> > +q->extco2.MaxQPP = q->max_qp_p > 51 ? 51 : q->max_qp_p;
> > +if (q->min_qp_b >= 0)
> > +q->extco2.MinQPB = q->min_qp_b > 51 ? 51 : q->min_qp_b;
> > +if (q->max_qp_b >= 0)
> > +q->extco2.MaxQPB = q->max_qp_b > 51 ? 51 : q->max_qp_b;
> >  #endif
> >  if (q->mbbrc >= 0)
> >  q->extco2.MBBRC = q->mbbrc ? MFX_CODINGOPTION_ON :
> > MFX_CODINGOPTION_OFF;
> > diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h
> > index cb84723dfa..ea05967db5 100644
> > --- a/libavcodec/qsvenc.h
> > +++ b/libavcodec/qsvenc.h
> > @@ -105,6 +105,12 @@
> >  { "low_power", "enable low power mode(experimental: many limitations
> by mfx
> > version, BRC modes, etc.)", OFFSET(qsv.low_power), AV_OPT_TYPE_BOOL,
> { .i64 =
> > -1}, -1, 1, VE},\
> >  { "dblk_idc", "This option disable deblocking. It has value in range
> > 0~2.",   OFFSET(qsv.dblk_idc),   AV_OPT_TYPE_INT,{ .i64 = 0
> > },   0,  2,  VE},\
> >  { "low_delay_brc",   "Allow to strictly obey avg frame size",
> > OFFSET(qsv.low_delay_brc),  AV_OPT_TYPE_BOOL,{ .i64 = -1 }, -1,  1,
> VE
> > }, \
> > +{ "max_qp_i", "Maximum video quantizer scale for I
> > frame",   OFFSET(qsv.max_qp_i),   AV_OPT_TYPE_INT, { .i64 = -1 },  -
> > 1,  51, VE}, \
> > +{ "min_qp_i", "Minimum video quantizer scale for I
> > frame",   OFFSET(qsv.min_q

Re: [FFmpeg-devel] [PATCH v2] libavcodec/cbs_av1: Add size check before parse obu

[Chen, Wenbin]

> On 29/03/2022 09:29, Wenbin Chen wrote:
> > cbs_av1_write_unit() check pbc size after parsing obu frame, and return
> > AVERROR(ENOSPC) if pbc is small. pbc will be reallocated and this obu
> > frame will be parsed again, but this may cause error because
> > CodedBitstreamAV1Context has already been updated, for example
> > ref_order_hint is updated and will not match the same obu frame. Now
> size
> > check is added before parsing obu frame to avoid this error.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >   libavcodec/cbs_av1.c | 6 +++---
> >   1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
> > index 1229480567..29e7bc16df 100644
> > --- a/libavcodec/cbs_av1.c
> > +++ b/libavcodec/cbs_av1.c
> > @@ -1075,6 +1075,9 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
> >   put_bits32(pbc, 0);
> >   }
> >
> > +if (8 * (unit->data_size + obu->obu_size) > put_bits_left(pbc))
> > +return AVERROR(ENOSPC);
> 
> unit->data_size is not usefully set when we are writing here (it might be the
> size of the old bitstream in editing cases, or it might just be zero).

Thank you for pointing this out. If data_size is unset this check wouldn't work 
and
the problem still occurs. I will try to find a better way to fix this.

> 
> > +
> >   td = NULL;
> >   start_pos = put_bits_count(pbc);
> >
> > @@ -1196,9 +1199,6 @@ static int
> cbs_av1_write_obu(CodedBitstreamContext *ctx,
> >   flush_put_bits(pbc);
> >   av_assert0(data_pos <= start_pos);
> >
> > -if (8 * obu->obu_size > put_bits_left(pbc))
> > -return AVERROR(ENOSPC);
> > -
> >   if (obu->obu_size > 0) {
> >   memmove(pbc->buf + data_pos,
> >   pbc->buf + start_pos, header_size);
> 
> So, this doesn't work?  The header hasn't been written that point, so you
> don't know if there is enough space for both the OBU header and the OBU
> data.
> 
> Having the check in both places would be fine (the newly-added one being a
> way to bail early when there definitely isn't enough space), but that wouldn't
> do what you want.

Ok, I will keep the both places in my next patch if I still fix issue in this 
way. 

> 
> I'm not sure what the right answer is here.  Do we need some way to unwind
> the written header?  The initial buffer size is 1MB and gets doubled each 
> time,
> so this is not going to be hit very often.

Unwinding header is an alternative way. I will check If it is possible.

This problem is rare. The problem occurs when I frame below buffer size but one 
P/B
frame in the gop is greater than buffer size.

Thanks
Wenbin

> 
> - Mark
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvdec.c: extract frame packing arrangement data

[Chen, Wenbin]

> On Tue, 2022-04-26 at 18:00 +0800, Wenbin Chen wrote:
> > Use h264_sei to parse SEI data got from MediaSDK. Extract frame
> > packing arrangement information from SEI data and config AVStereo3D
> > side data for decoded frame.
> >
> > Signed-off-by: Wenbin Chen 
> > Signed-off-by: Tong Wu 
> > ---
> >  libavcodec/qsv_internal.h |   2 +
> >  libavcodec/qsvdec.c   | 160
> ++
> >  2 files changed, 162 insertions(+)
> >
> > diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> > index e2aecdcbd6..a804c392c1 100644
> > --- a/libavcodec/qsv_internal.h
> > +++ b/libavcodec/qsv_internal.h
> > @@ -54,6 +54,8 @@
> >
> >  #define QSV_MAX_FRAME_EXT_PARAMS 4
> >
> > +#define QSV_PAYLOAD_SIZE 1024
> > +
> >  #define QSV_VERSION_ATLEAST(MAJOR, MINOR)   \
> >  (MFX_VERSION_MAJOR > (MAJOR) || \
> >   MFX_VERSION_MAJOR == (MAJOR) && MFX_VERSION_MINOR >=
> (MINOR))
> > diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
> > index 5fc5bed4c8..26fa178b4d 100644
> > --- a/libavcodec/qsvdec.c
> > +++ b/libavcodec/qsvdec.c
> > @@ -41,13 +41,16 @@
> >  #include "libavutil/time.h"
> >  #include "libavutil/imgutils.h"
> >  #include "libavutil/film_grain_params.h"
> > +#include "libavutil/stereo3d.h"
> >
> >  #include "avcodec.h"
> >  #include "codec_internal.h"
> >  #include "internal.h"
> >  #include "decode.h"
> >  #include "hwconfig.h"
> > +#include "get_bits.h"
> >  #include "qsv.h"
> > +#include "h264_sei.h"
> >  #include "qsv_internal.h"
> >
> >  static const AVRational mfx_tb = { 1, 9 };
> > @@ -101,6 +104,10 @@ typedef struct QSVContext {
> >
> >  mfxExtBuffer **ext_buffers;
> >  int nb_ext_buffers;
> > +
> > +mfxPayload payload;
> > +H264SEIContext sei;
> > +H264ParamSets ps;
> >  } QSVContext;
> >
> >  static const AVCodecHWConfigInternal *const qsv_hw_configs[] = {
> > @@ -600,6 +607,150 @@ static int qsv_export_film_grain(AVCodecContext
> *avctx,
> > mfxExtAV1FilmGrainParam
> >  }
> >  #endif
> >
> > +static int h264_decode_fpa(H264SEIFramePacking *fpa, AVFrame *frame)
> > +{
> > +if (!fpa || !frame)
> > +return AVERROR(EINVAL);
> > +
> > +if (!fpa->arrangement_cancel_flag &&
> > +fpa->arrangement_type <= 6 &&
> > +fpa->content_interpretation_type > 0 &&
> > +fpa->content_interpretation_type < 3) {
> > +AVStereo3D *stereo = av_stereo3d_create_side_data(frame);
> > +if (stereo) {
> > +switch (fpa->arrangement_type) {
> > +case 0:
> > +stereo->type = AV_STEREO3D_CHECKERBOARD;
> > +break;
> > +case 1:
> > +stereo->type = AV_STEREO3D_COLUMNS;
> > +break;
> > +case 2:
> > +stereo->type = AV_STEREO3D_LINES;
> > +break;
> > +case 3:
> > +if (fpa->quincunx_sampling_flag)
> > +stereo->type = AV_STEREO3D_SIDEBYSIDE_QUINCUNX;
> > +else
> > +stereo->type = AV_STEREO3D_SIDEBYSIDE;
> > +break;
> > +case 4:
> > +stereo->type = AV_STEREO3D_TOPBOTTOM;
> > +break;
> > +case 5:
> > +stereo->type = AV_STEREO3D_FRAMESEQUENCE;
> > +if (fpa->current_frame_is_frame0_flag)
> > +stereo->view = AV_STEREO3D_VIEW_LEFT;
> > +else
> > +stereo->view = AV_STEREO3D_VIEW_RIGHT;
> > +break;
> > +case 6:
> > +stereo->type = AV_STEREO3D_2D;
> > +break;
> > +}
> > +
> > +if (fpa->content_interpretation_type == 2)
> > +stereo->flags = AV_STEREO3D_FLAG_INVERT;
> > +}
> > +}
> > +return 0;
> > +}
> > +
> > +static int h264_parse_side_data(AVCodecContext *avctx, QSVContext *q,
> AVFrame
> > *frame)
> > +{
> > +GetBitContext gb_payload;
> > +uint8_t *sei_buffer;
> > +int sei_buffer_index;
> > +int ret;
> > +
> > +if (q->payload.Type != SEI_TYPE_FRAME_PACKING_ARRANGEMENT)
> > +return 0;
> > +
> > +sei_buffer = (uint8_t *)av_mallocz(q->payload.NumBit / 8);
> > +if (!sei_buffer) {
> > +av_freep(&sei_buffer);
> > +return AVERROR(ENOMEM);
> > +}
> > +/* remove emulation prevention bytes */
> > +sei_buffer_index = 0;
> > +for (int i = 0; i < q->payload.NumBit / 8; i++) {
> > +if (q->payload.Data[i] == 3)
> > +i++;
> > +sei_buffer[sei_buffer_index] = q->payload.Data[i];
> > +sei_buffer_index += 1;
> > +}
> > +
> > +ret = init_get_bits8(&gb_payload, sei_buffer, sei_buffer_index+1);
> > +if (ret < 0) {
> > +av_freep(&sei_buffer);
> > +return ret;
> > +}
> > +
> > +ret = ff_h264_sei_decode(&q->sei, &gb_payload, &q->ps, avctx);
> > +if (ret < 0) {
> > +av_freep(&sei_buff

Re: [FFmpeg-devel] [PATCH v2 1/2] libavcodec/qsvdec: Add more pixel format support to qsvdec

[Chen, Wenbin]

> On Sat, 2022-04-02 at 17:35 +0800, Wenbin Chen wrote:
> > Qsv decoder only supports directly output nv12 and p010 to system
> > memory. For other format, we need to download frame from qsv format
> > to system memory. Now add other supported format to qsvdec.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/qsv.c  | 36 
> >  libavcodec/qsv_internal.h |  3 +++
> >  libavcodec/qsvdec.c   | 23 +--
> >  3 files changed, 56 insertions(+), 6 deletions(-)
> >
> > diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
> > index b75877e698..cc1352aa2a 100644
> > --- a/libavcodec/qsv.c
> > +++ b/libavcodec/qsv.c
> > @@ -244,6 +244,42 @@ int ff_qsv_map_pixfmt(enum AVPixelFormat
> format, uint32_t
> > *fourcc)
> >  }
> >  }
> >
> > +int ff_qsv_map_frame_to_surface(const AVFrame *frame,
> mfxFrameSurface1
> > *surface)
> > +{
> > +switch (frame->format) {
> > +case AV_PIX_FMT_NV12:
> > +case AV_PIX_FMT_P010:
> > +surface->Data.Y  = frame->data[0];
> > +surface->Data.UV = frame->data[1];
> > +/* The SDK checks Data.V when using system memory for VP9
> encoding */
> > +surface->Data.V  = surface->Data.UV + 1;
> > +break;
> > +case AV_PIX_FMT_X2RGB10LE:
> > +case AV_PIX_FMT_BGRA:
> > +surface->Data.B = frame->data[0];
> > +surface->Data.G = frame->data[0] + 1;
> > +surface->Data.R = frame->data[0] + 2;
> > +surface->Data.A = frame->data[0] + 3;
> > +break;
> > +case AV_PIX_FMT_YUYV422:
> > +surface->Data.Y = frame->data[0];
> > +surface->Data.U = frame->data[0] + 1;
> > +surface->Data.V = frame->data[0] + 3;
> > +break;
> > +
> > +case AV_PIX_FMT_Y210:
> > +surface->Data.Y16 = (mfxU16 *)frame->data[0];
> > +surface->Data.U16 = (mfxU16 *)frame->data[0] + 1;
> > +surface->Data.V16 = (mfxU16 *)frame->data[0] + 3;
> > +break;
> > +default:
> > +return AVERROR(ENOSYS);
> > +}
> > +surface->Data.PitchLow  = frame->linesize[0];
> > +
> > +return 0;
> > +}
> > +
> >  int ff_qsv_find_surface_idx(QSVFramesContext *ctx, QSVFrame *frame)
> >  {
> >  int i;
> > diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> > index 58186ea7ca..e2aecdcbd6 100644
> > --- a/libavcodec/qsv_internal.h
> > +++ b/libavcodec/qsv_internal.h
> > @@ -147,4 +147,7 @@ int ff_qsv_find_surface_idx(QSVFramesContext
> *ctx,
> > QSVFrame *frame);
> >  void ff_qsv_frame_add_ext_param(AVCodecContext *avctx, QSVFrame
> *frame,
> >  mfxExtBuffer *param);
> >
> > +int ff_qsv_map_frame_to_surface(const AVFrame *frame,
> mfxFrameSurface1
> > *surface);
> > +
> > +
> >  #endif /* AVCODEC_QSV_INTERNAL_H */
> > diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
> > index 6236391357..f1d56b2af3 100644
> > --- a/libavcodec/qsvdec.c
> > +++ b/libavcodec/qsvdec.c
> > @@ -129,21 +129,28 @@ static int
> qsv_get_continuous_buffer(AVCodecContext
> > *avctx, AVFrame *frame,
> >  frame->linesize[0] = FFALIGN(avctx->width, 128);
> >  break;
> >  case AV_PIX_FMT_P010:
> > +case AV_PIX_FMT_YUYV422:
> >  frame->linesize[0] = 2 * FFALIGN(avctx->width, 128);
> >  break;
> > +case AV_PIX_FMT_Y210:
> > +frame->linesize[0] = 4 * FFALIGN(avctx->width, 128);
> > +break;
> >  default:
> >  av_log(avctx, AV_LOG_ERROR, "Unsupported pixel format.\n");
> >  return AVERROR(EINVAL);
> >  }
> >
> > -frame->linesize[1] = frame->linesize[0];
> >  frame->buf[0]  = av_buffer_pool_get(pool);
> >  if (!frame->buf[0])
> >  return AVERROR(ENOMEM);
> >
> >  frame->data[0] = frame->buf[0]->data;
> > -frame->data[1] = frame->data[0] +
> > -frame->linesize[0] * FFALIGN(avctx->height, 
> > 64);
> > +if (avctx->pix_fmt == AV_PIX_FMT_NV12 ||
> > +avctx->pix_fmt == AV_PIX_FMT_P010) {
> > +frame->linesize[1] = frame->linesize[0];
> > +frame->data[1] = frame->data[0] +
> > +frame->linesize[0] * FFALIGN(avctx->height, 64);
> > +}
> >
> >  ret = ff_attach_decode_data(frame);
> >  if (ret < 0)
> > @@ -423,9 +430,11 @@ static int alloc_frame(AVCodecContext *avctx,
> QSVContext
> > *q, QSVFrame *frame)
> >  if (frame->frame->format == AV_PIX_FMT_QSV) {
> >  frame->surface = *(mfxFrameSurface1*)frame->frame->data[3];
> >  } else {
> > -frame->surface.Data.PitchLow = frame->frame->linesize[0];
> > -frame->surface.Data.Y= frame->frame->data[0];
> > -frame->surface.Data.UV   = frame->frame->data[1];
> > +ret = ff_qsv_map_frame_to_surface(frame->frame, &frame-
> >surface);
> > +if (ret < 0) {
> > +av_log(avctx, AV_LOG_ERROR, "map frame to surface failed.\n");
> > +return ret;
> > +}
> >  }
> >
> >  frame->

Re: [FFmpeg-devel] [PATCH 1/2] libavcodec/qsvdec: Add more pixel format support to qsvdec

[Chen, Wenbin]

> On Wed, 2022-03-30 at 14:43 +0800, Wenbin Chen wrote:
> > Qsv decoder only supports directly output nv12 and p010 to system
> > memory. For other format, we need to download frame from qsv format
> > to system memory. Now add other supported format to qsvdec.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/qsv.c  | 36 
> >  libavcodec/qsv_internal.h |  3 +++
> >  libavcodec/qsvdec.c   | 23 +--
> >  3 files changed, 56 insertions(+), 6 deletions(-)
> >
> > diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
> > index 67d0e3934a..8010eef172 100644
> > --- a/libavcodec/qsv.c
> > +++ b/libavcodec/qsv.c
> > @@ -244,6 +244,42 @@ int ff_qsv_map_pixfmt(enum AVPixelFormat
> format, uint32_t
> > *fourcc)
> >  }
> >  }
> >
> > +int ff_qsv_map_frame_to_surface(const AVFrame *frame,
> mfxFrameSurface1
> > *surface)
> > +{
> > +switch (frame->format) {
> > +case AV_PIX_FMT_NV12:
> > +case AV_PIX_FMT_P010:
> > +surface->Data.Y  = frame->data[0];
> > +surface->Data.UV = frame->data[1];
> > +/* The SDK checks Data.V when using system memory for VP9
> encoding */
> > +surface->Data.V  = surface->Data.UV + 1;
> > +break;
> > +case AV_PIX_FMT_X2RGB10LE:
> > +case AV_PIX_FMT_BGRA:
> > +surface->Data.B = frame->data[0];
> > +surface->Data.G = frame->data[0] + 1;
> > +surface->Data.R = frame->data[0] + 2;
> > +surface->Data.A = frame->data[0] + 3;
> > +break;
> > +case AV_PIX_FMT_YUYV422:
> > +surface->Data.Y = frame->data[0];
> > +surface->Data.U = frame->data[0] + 1;
> > +surface->Data.V = frame->data[0] + 3;
> > +break;
> > +
> > +case AV_PIX_FMT_Y210:
> > +surface->Data.Y16 = (mfxU16 *)frame->data[0];
> > +surface->Data.U16 = (mfxU16 *)frame->data[0] + 1;
> > +surface->Data.V16 = (mfxU16 *)frame->data[0] + 3;
> > +break;
> > +default:
> > +return MFX_ERR_UNSUPPORTED;
> 
> Please change the return type to mfxStatus if you want to return a mfx error
> code, otherwise return a ffmpeg error here.
> 
> Thanks
> Haihao
> 

Thanks for review. I will update it.

> 
> > +}
> > +surface->Data.PitchLow  = frame->linesize[0];
> > +
> > +return 0;
> > +}
> > +
> >  int ff_qsv_find_surface_idx(QSVFramesContext *ctx, QSVFrame *frame)
> >  {
> >  int i;
> > diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> > index 58186ea7ca..e2aecdcbd6 100644
> > --- a/libavcodec/qsv_internal.h
> > +++ b/libavcodec/qsv_internal.h
> > @@ -147,4 +147,7 @@ int ff_qsv_find_surface_idx(QSVFramesContext
> *ctx,
> > QSVFrame *frame);
> >  void ff_qsv_frame_add_ext_param(AVCodecContext *avctx, QSVFrame
> *frame,
> >  mfxExtBuffer *param);
> >
> > +int ff_qsv_map_frame_to_surface(const AVFrame *frame,
> mfxFrameSurface1
> > *surface);
> > +
> > +
> >  #endif /* AVCODEC_QSV_INTERNAL_H */
> > diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
> > index 661f15bc75..f159e2690f 100644
> > --- a/libavcodec/qsvdec.c
> > +++ b/libavcodec/qsvdec.c
> > @@ -132,21 +132,28 @@ static int
> qsv_get_continuous_buffer(AVCodecContext
> > *avctx, AVFrame *frame,
> >  frame->linesize[0] = FFALIGN(avctx->width, 128);
> >  break;
> >  case AV_PIX_FMT_P010:
> > +case AV_PIX_FMT_YUYV422:
> >  frame->linesize[0] = 2 * FFALIGN(avctx->width, 128);
> >  break;
> > +case AV_PIX_FMT_Y210:
> > +frame->linesize[0] = 4 * FFALIGN(avctx->width, 128);
> > +break;
> >  default:
> >  av_log(avctx, AV_LOG_ERROR, "Unsupported pixel format.\n");
> >  return AVERROR(EINVAL);
> >  }
> >
> > -frame->linesize[1] = frame->linesize[0];
> >  frame->buf[0]  = av_buffer_pool_get(pool);
> >  if (!frame->buf[0])
> >  return AVERROR(ENOMEM);
> >
> >  frame->data[0] = frame->buf[0]->data;
> > -frame->data[1] = frame->data[0] +
> > -frame->linesize[0] * FFALIGN(avctx->height, 
> > 64);
> > +if (avctx->pix_fmt == AV_PIX_FMT_NV12 ||
> > +avctx->pix_fmt == AV_PIX_FMT_P010) {
> > +frame->linesize[1] = frame->linesize[0];
> > +frame->data[1] = frame->data[0] +
> > +frame->linesize[0] * FFALIGN(avctx->height, 64);
> > +}
> >
> >  ret = ff_attach_decode_data(frame);
> >  if (ret < 0)
> > @@ -426,9 +433,11 @@ static int alloc_frame(AVCodecContext *avctx,
> QSVContext
> > *q, QSVFrame *frame)
> >  if (frame->frame->format == AV_PIX_FMT_QSV) {
> >  frame->surface = *(mfxFrameSurface1*)frame->frame->data[3];
> >  } else {
> > -frame->surface.Data.PitchLow = frame->frame->linesize[0];
> > -frame->surface.Data.Y= frame->frame->data[0];
> > -frame->surface.Data.UV   = frame->frame->data[1];
> > +ret = ff_qsv_map_frame_to_surface(frame->frame, &frame

Re: [FFmpeg-devel] [PATCH v2] libavcodec/qsvenc_hevc: encode RGB format rawvideo

[Chen, Wenbin]

> On Tue, 2022-02-15 at 15:00 +0800, Wenbin Chen wrote:
> > Add support for hevc_qsv to input RGB format frame. It will
> > transform frame to yuv inside MediaSDK instead of using auto
> > scale. Now hevc_qsv supports directly encoding BGRA and X2RGB10
> > format. X2RGB10 is only supported in VDENC (-low_power 1).
> > The X2RGB10 correspond to the A2RGB20 format in MediaSDK.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/qsv.c | 16 
> >  libavcodec/qsvenc.c  | 13 +
> >  libavcodec/qsvenc_hevc.c |  6 ++
> >  3 files changed, 35 insertions(+)
> >
> > diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
> > index 1a432dbd82..b75877e698 100644
> > --- a/libavcodec/qsv.c
> > +++ b/libavcodec/qsv.c
> > @@ -189,6 +189,12 @@ enum AVPixelFormat ff_qsv_map_fourcc(uint32_t
> fourcc)
> >  case MFX_FOURCC_NV12: return AV_PIX_FMT_NV12;
> >  case MFX_FOURCC_P010: return AV_PIX_FMT_P010;
> >  case MFX_FOURCC_P8:   return AV_PIX_FMT_PAL8;
> > +#if QSV_VERSION_ATLEAST(1, 9)
> > +case MFX_FOURCC_A2RGB10: return AV_PIX_FMT_X2RGB10;
> > +#endif
> > +#if QSV_VERSION_ATLEAST(1, 17)
> > +case MFX_FOURCC_RGB4: return AV_PIX_FMT_BGRA;
> > +#endif
> >  #if CONFIG_VAAPI
> >  case MFX_FOURCC_YUY2: return AV_PIX_FMT_YUYV422;
> >  #if QSV_VERSION_ATLEAST(1, 27)
> > @@ -211,6 +217,16 @@ int ff_qsv_map_pixfmt(enum AVPixelFormat
> format, uint32_t
> > *fourcc)
> >  case AV_PIX_FMT_P010:
> >  *fourcc = MFX_FOURCC_P010;
> >  return AV_PIX_FMT_P010;
> > +#if QSV_VERSION_ATLEAST(1, 9)
> > +case AV_PIX_FMT_X2RGB10:
> > +*fourcc = MFX_FOURCC_A2RGB10;
> > +return AV_PIX_FMT_X2RGB10;
> > +#endif
> > +#if QSV_VERSION_ATLEAST(1, 17)
> > +case AV_PIX_FMT_BGRA:
> > +*fourcc = MFX_FOURCC_RGB4;
> > +return AV_PIX_FMT_BGRA;
> > +#endif
> >  #if CONFIG_VAAPI
> >  case AV_PIX_FMT_YUV422P:
> >  case AV_PIX_FMT_YUYV422:
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index 07be4287b7..104133fc59 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -715,6 +715,11 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  if (ret < 0)
> >  return AVERROR_BUG;
> >
> > +if (sw_format == AV_PIX_FMT_X2RGB10 && q->low_power != 1) {
> 
> The SDK may choose a workable mode if low_power is set to auto.
> 
> Thanks
> Haihao

Thanks for your information. I will update patch.

> 
> > +av_log(avctx, AV_LOG_ERROR, "Only VDENC support encoding
> x2rgb10\n");
> > +return AVERROR(EINVAL);
> > +}
> > +
> >  q->param.mfx.FrameInfo.CropX  = 0;
> >  q->param.mfx.FrameInfo.CropY  = 0;
> >  q->param.mfx.FrameInfo.CropW  = avctx->width;
> > @@ -1616,6 +1621,14 @@ static int submit_frame(QSVEncContext *q,
> const AVFrame
> > *frame,
> >  qf->surface.Data.V = qf->surface.Data.UV + 2;
> >  break;
> >
> > +case AV_PIX_FMT_X2RGB10:
> > +case AV_PIX_FMT_BGRA:
> > +qf->surface.Data.B = qf->frame->data[0];
> > +qf->surface.Data.G = qf->frame->data[0] + 1;
> > +qf->surface.Data.R = qf->frame->data[0] + 2;
> > +qf->surface.Data.A = qf->frame->data[0] + 3;
> > +break;
> > +
> >  default:
> >  /* should not reach here */
> >  av_assert0(0);
> > diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c
> > index 5cac141c4d..ade546d4ca 100644
> > --- a/libavcodec/qsvenc_hevc.c
> > +++ b/libavcodec/qsvenc_hevc.c
> > @@ -304,6 +304,12 @@ const AVCodec ff_hevc_qsv_encoder = {
> >  .pix_fmts   = (const enum AVPixelFormat[]){ AV_PIX_FMT_NV12,
> >  AV_PIX_FMT_P010,
> >  AV_PIX_FMT_QSV,
> > +#if QSV_VERSION_ATLEAST(1, 17)
> > +AV_PIX_FMT_BGRA,
> > +#endif
> > +#if QSV_VERSION_ATLEAST(1, 9)
> > +AV_PIX_FMT_X2RGB10,
> > +#endif
> >  AV_PIX_FMT_NONE },
> >  .priv_class = &class,
> >  .defaults   = qsv_enc_defaults,
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/qsvenc_hevc: encode RGB format rawvideo

[Chen, Wenbin]

> Wenbin Chen:
> > Add support for hevc_qsv to input RGB format frame. It will
> > transform frame to yuv inside MediaSDK instead of using auto
> > scale. Now hevc_qsv supports directly encoding BGRA and X2RGB10
> > format. X2RGB10 is only supported in VDENC (-low_power 1).
> > The X2RGB10 correspond to the A2RGB20 format in MediaSDK.
> >
> > Sigend-off-by: Wenbin Chen 
>  ^
> Are you writing this manually?

I always thought it is manually writing. I just searched and found that
there is  a "-s" option. Will fix this.
Thanks for your reminder.

> 
> - Andreas
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V3 1/3] libavcodec/vaapi_encode: Add new API adaption to vaapi_encode

[Chen, Wenbin]

> On Tue, 2022-02-08 at 11:05 +0800, Wenbin Chen wrote:
> > Add vaSyncBuffer to VAAPI encoder. Old version API vaSyncSurface wait
> > surface to complete. When surface is used for multiple operation, it
> > waits all operations to finish. vaSyncBuffer only wait one channel to
> > finish.
> >
> > Add wait param to vaapi_encode_wait() to prepare for the async_depth
> > option. "wait=1" means wait until operation ready. "wait=0" means
> > query operation's status. If it is ready return 0, if it is still
> > in progress return EAGAIN.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/vaapi_encode.c | 47 +---
> ---
> >  1 file changed, 40 insertions(+), 7 deletions(-)
> >
> > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> > index 3bf379b1a0..b87b58a42b 100644
> > --- a/libavcodec/vaapi_encode.c
> > +++ b/libavcodec/vaapi_encode.c
> > @@ -134,7 +134,8 @@ static int
> > vaapi_encode_make_misc_param_buffer(AVCodecContext *avctx,
> >  }
> >
> >  static int vaapi_encode_wait(AVCodecContext *avctx,
> > - VAAPIEncodePicture *pic)
> > + VAAPIEncodePicture *pic,
> > + uint8_t wait)
> >  {
> >  VAAPIEncodeContext *ctx = avctx->priv_data;
> >  VAStatus vas;
> > @@ -150,11 +151,43 @@ static int vaapi_encode_wait(AVCodecContext
> *avctx,
> > "(input surface %#x).\n", pic->display_order,
> > pic->encode_order, pic->input_surface);
> >
> > -vas = vaSyncSurface(ctx->hwctx->display, pic->input_surface);
> > -if (vas != VA_STATUS_SUCCESS) {
> > -av_log(avctx, AV_LOG_ERROR, "Failed to sync to picture completion: 
> > "
> > -   "%d (%s).\n", vas, vaErrorStr(vas));
> > +#if VA_CHECK_VERSION(1, 9, 0)
> > +// Try vaSyncBuffer.
> > +vas = vaSyncBuffer(ctx->hwctx->display,
> > +   pic->output_buffer,
> > +   wait ? VA_TIMEOUT_INFINITE : 0);
> > +if (vas == VA_STATUS_ERROR_TIMEDOUT) {
> > +return AVERROR(EAGAIN);
> > +} else if (vas != VA_STATUS_SUCCESS && vas !=
> > VA_STATUS_ERROR_UNIMPLEMENTED) {
> > +av_log(avctx, AV_LOG_ERROR, "Failed to sync to output buffer
> > completion: "
> > +"%d (%s).\n", vas, vaErrorStr(vas));
> >  return AVERROR(EIO);
> 
> We may add has_sync_buffer_func flag in this patch, and run the above code
> when
> ctx->has_sync_buffer_func is true. If so, we needn't check whether
> vaSyncBuffer
> is implemented again.
> 
> Thanks
> Haihao

Thanks for your review. I will update the patchset.

Thanks
Wenbin
> 
> 
> > +} else if (vas == VA_STATUS_ERROR_UNIMPLEMENTED)
> > +// If vaSyncBuffer is not implemented, try old version API.
> > +#endif
> > +{
> > +if (!wait) {
> > +VASurfaceStatus surface_status;
> > +vas = vaQuerySurfaceStatus(ctx->hwctx->display,
> > +pic->input_surface,
> > +&surface_status);
> > +if (vas == VA_STATUS_SUCCESS &&
> > +surface_status != VASurfaceReady &&
> > +surface_status != VASurfaceSkipped) {
> > +return AVERROR(EAGAIN);
> > +} else if (vas != VA_STATUS_SUCCESS) {
> > +av_log(avctx, AV_LOG_ERROR, "Failed to query surface 
> > status:
> > "
> > +"%d (%s).\n", vas, vaErrorStr(vas));
> > +return AVERROR(EIO);
> > +}
> > +} else {
> > +vas = vaSyncSurface(ctx->hwctx->display, pic->input_surface);
> > +if (vas != VA_STATUS_SUCCESS) {
> > +av_log(avctx, AV_LOG_ERROR, "Failed to sync to picture
> > completion: "
> > +"%d (%s).\n", vas, vaErrorStr(vas));
> > +return AVERROR(EIO);
> > +}
> > +}
> >  }
> >
> >  // Input is definitely finished with now.
> > @@ -633,7 +666,7 @@ static int vaapi_encode_output(AVCodecContext
> *avctx,
> >  uint8_t *ptr;
> >  int err;
> >
> > -err = vaapi_encode_wait(avctx, pic);
> > +err = vaapi_encode_wait(avctx, pic, 1);
> >  if (err < 0)
> >  return err;
> >
> > @@ -695,7 +728,7 @@ fail:
> >  static int vaapi_encode_discard(AVCodecContext *avctx,
> >  VAAPIEncodePicture *pic)
> >  {
> > -vaapi_encode_wait(avctx, pic);
> > +vaapi_encode_wait(avctx, pic, 1);
> >
> >  if (pic->output_buffer_ref) {
> >  av_log(avctx, AV_LOG_DEBUG, "Discard output for pic "
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailma

Re: [FFmpeg-devel] [PATCH V3 3/3] libavcodec/vaapi_encode: Add async_depth to vaapi_encoder to increase performance

[Chen, Wenbin]

> Add async_depth to increase encoder's performance. Reuse encode_fifo as
> async buffer. Encoder puts all reordered frame to HW and then check
> fifo size. If fifo < async_depth and the top frame is not ready, it will
> return AVERROR(EAGAIN) to require more frames.
> 
> 1080p transcoding (no B frames) with -async_depth=4 can increase 20%
> performance on my environment.
> The async increases performance but also introduces frame delay.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/vaapi_encode.c | 16 
>  libavcodec/vaapi_encode.h | 12 ++--
>  2 files changed, 22 insertions(+), 6 deletions(-)
> 
> diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> index 15ddbbaa4a..432abf31f7 100644
> --- a/libavcodec/vaapi_encode.c
> +++ b/libavcodec/vaapi_encode.c
> @@ -1158,7 +1158,8 @@ static int
> vaapi_encode_send_frame(AVCodecContext *avctx, AVFrame *frame)
>  if (ctx->input_order == ctx->decode_delay)
>  ctx->dts_pts_diff = pic->pts - ctx->first_pts;
>  if (ctx->output_delay > 0)
> -ctx->ts_ring[ctx->input_order % (3 * ctx->output_delay)] = 
> pic->pts;
> +ctx->ts_ring[ctx->input_order %
> +(3 * ctx->output_delay + ctx->async_depth)] = 
> pic->pts;
> 
>  pic->display_order = ctx->input_order;
>  ++ctx->input_order;
> @@ -1214,7 +1215,7 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
> 
>  #if VA_CHECK_VERSION(1, 9, 0)
>  if (ctx->has_sync_buffer_func) {
> -while (av_fifo_can_read(ctx->encode_fifo) <=
> MAX_PICTURE_REFERENCES) {
> +while (av_fifo_can_read(ctx->encode_fifo) <= MAX_ASYNC_DEPTH) {

Here is a mistake I should use "<" instead of "<=" and I can use 
av_fifo_can_write()
instead. I will update it.

>  pic = NULL;
>  err = vaapi_encode_pick_next(avctx, &pic);
>  if (err < 0)
> @@ -1232,6 +1233,13 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
>  }
>  if (!av_fifo_can_read(ctx->encode_fifo))
>  return err;
> +if (av_fifo_can_read(ctx->encode_fifo) < ctx->async_depth &&
> +!ctx->end_of_stream) {
> +av_fifo_peek(ctx->encode_fifo, &pic, 1, 0);
> +err = vaapi_encode_wait(avctx, pic, 0);
> +if (err < 0)
> +return err;
> +}
>  av_fifo_read(ctx->encode_fifo, &pic, 1);
>  ctx->encode_order = pic->encode_order + 1;
>  } else
> @@ -1267,7 +1275,7 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
>  pkt->dts = ctx->ts_ring[pic->encode_order] - ctx->dts_pts_diff;
>  } else {
>  pkt->dts = ctx->ts_ring[(pic->encode_order - ctx->decode_delay) %
> -(3 * ctx->output_delay)];
> +(3 * ctx->output_delay + ctx->async_depth)];
>  }
>  av_log(avctx, AV_LOG_DEBUG, "Output packet: pts %"PRId64"
> dts %"PRId64".\n",
> pkt->pts, pkt->dts);
> @@ -2588,7 +2596,7 @@ av_cold int ff_vaapi_encode_init(AVCodecContext
> *avctx)
>  vas = vaSyncBuffer(ctx->hwctx->display, 0, 0);
>  if (vas != VA_STATUS_ERROR_UNIMPLEMENTED) {
>  ctx->has_sync_buffer_func = 1;
> -ctx->encode_fifo = av_fifo_alloc2(MAX_PICTURE_REFERENCES + 1,
> +ctx->encode_fifo = av_fifo_alloc2(MAX_ASYNC_DEPTH,
>sizeof(VAAPIEncodePicture *),
>0);
>  if (!ctx->encode_fifo)
> diff --git a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h
> index d33a486cb8..691521387d 100644
> --- a/libavcodec/vaapi_encode.h
> +++ b/libavcodec/vaapi_encode.h
> @@ -48,6 +48,7 @@ enum {
>  MAX_TILE_ROWS  = 22,
>  // A.4.1: table A.6 allows at most 20 tile columns for any level.
>  MAX_TILE_COLS  = 20,
> +MAX_ASYNC_DEPTH= 64,
>  };
> 
>  extern const AVCodecHWConfigInternal *const
> ff_vaapi_encode_hw_configs[];
> @@ -298,7 +299,8 @@ typedef struct VAAPIEncodeContext {
>  // Timestamp handling.
>  int64_t first_pts;
>  int64_t dts_pts_diff;
> -int64_t ts_ring[MAX_REORDER_DELAY * 3];
> +int64_t ts_ring[MAX_REORDER_DELAY * 3 +
> +MAX_ASYNC_DEPTH];
> 
>  // Slice structure.
>  int slice_block_rows;
> @@ -350,6 +352,8 @@ typedef struct VAAPIEncodeContext {
>  AVFifo *encode_fifo;
>  //Whether the driver support vaSyncBuffer
>  int has_sync_buffer_func;
> +//Max number of frame buffered in encoder.
> +int async_depth;
>  } VAAPIEncodeContext;
> 
>  enum {
> @@ -460,7 +464,11 @@ int ff_vaapi_encode_close(AVCodecContext *avctx);
>  { "b_depth", \
>"Maximum B-frame reference depth", \
>OFFSET(common.desired_b_depth), AV_OPT_TYPE_INT, \
> -  { .i64 = 1 }, 1, INT_MAX, FLAGS }
> +  { .i64 = 1 }, 1,

Re: [FFmpeg-devel] [PATCH V2 2/3] libavcodec/vaapi_encode: Change the way to call async to increase performance

[Chen, Wenbin]

> On Wed, 2022-01-05 at 10:48 +0800, Wenbin Chen wrote:
> > Fix: #7706. After commit 5fdcf85bbffe7451c2, vaapi encoder's performance
> > decrease. The reason is that vaRenderPicture() and vaSyncBuffer() are
> > called at the same time (vaRenderPicture() always followed by a
> > vaSyncBuffer()). When we encode stream with B frames, we need buffer to
> > reorder frames, so we can send serveral frames to HW at once to increase
> > performance. Now I changed them to be called in a asynchronous way,
> which
> > will make better use of hardware. 1080p transcoding increases about 17%
> > fps on my environment.
> >
> > This change fits vaSyncBuffer(), so if driver does not support
> > vaSyncBuffer, it will keep previous operation.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/vaapi_encode.c | 64 -
> --
> >  libavcodec/vaapi_encode.h |  5 +++
> >  2 files changed, 58 insertions(+), 11 deletions(-)
> >
> > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> > index b87b58a42b..9a3b3ba4ad 100644
> > --- a/libavcodec/vaapi_encode.c
> > +++ b/libavcodec/vaapi_encode.c
> > @@ -984,8 +984,10 @@ static int
> vaapi_encode_pick_next(AVCodecContext *avctx,
> >  if (!pic && ctx->end_of_stream) {
> >  --b_counter;
> >  pic = ctx->pic_end;
> > -if (pic->encode_issued)
> > +if (pic->encode_complete)
> >  return AVERROR_EOF;
> > +else if (pic->encode_issued)
> > +return AVERROR(EAGAIN);
> >  }
> >
> >  if (!pic) {
> > @@ -1210,18 +1212,45 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext
> > *avctx, AVPacket *pkt)
> >  return AVERROR(EAGAIN);
> >  }
> >
> > -pic = NULL;
> > -err = vaapi_encode_pick_next(avctx, &pic);
> > -if (err < 0)
> > -return err;
> > -av_assert0(pic);
> > +#if VA_CHECK_VERSION(1, 9, 0)
> > +if (ctx->has_sync_buffer_func) {
> > +while (av_fifo_size(ctx->encode_fifo) <=
> > +   MAX_PICTURE_REFERENCES * sizeof(VAAPIEncodePicture *)) {
> > +pic = NULL;
> > +err = vaapi_encode_pick_next(avctx, &pic);
> > +if (err < 0)
> > +break;
> > +
> > +av_assert0(pic);
> > +pic->encode_order = ctx->encode_order +
> > +(av_fifo_size(ctx->encode_fifo) / sizeof(VAAPIEncodePicture
> > *));
> > +err = vaapi_encode_issue(avctx, pic);
> > +if (err < 0) {
> > +av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > +return err;
> > +}
> > +av_fifo_generic_write(ctx->encode_fifo, &pic, sizeof(pic), 
> > NULL);
> > +}
> > +if (!av_fifo_size(ctx->encode_fifo))
> > +return err;
> > +av_fifo_generic_read(ctx->encode_fifo, &pic, sizeof(pic), NULL);
> > +ctx->encode_order = pic->encode_order + 1;
> > +} else
> > +#endif
> > +{
> > +pic = NULL;
> > +err = vaapi_encode_pick_next(avctx, &pic);
> > +if (err < 0)
> > +return err;
> > +av_assert0(pic);
> >
> > -pic->encode_order = ctx->encode_order++;
> > +pic->encode_order = ctx->encode_order++;
> >
> > -err = vaapi_encode_issue(avctx, pic);
> > -if (err < 0) {
> > -av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > -return err;
> > +err = vaapi_encode_issue(avctx, pic);
> > +if (err < 0) {
> > +av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > +return err;
> > +}
> >  }
> >
> >  err = vaapi_encode_output(avctx, pic, pkt);
> > @@ -2555,6 +2584,18 @@ av_cold int
> ff_vaapi_encode_init(AVCodecContext *avctx)
> >  }
> >  }
> >
> > +#if VA_CHECK_VERSION(1, 9, 0)
> > +//check vaSyncBuffer function
> > +vas = vaSyncBuffer(ctx->hwctx->display, 0, 0);
> > +if (vas != VA_STATUS_ERROR_UNIMPLEMENTED) {
> > +ctx->has_sync_buffer_func = 1;
> > +ctx->encode_fifo = av_fifo_alloc((MAX_PICTURE_REFERENCES + 1) *
> > +sizeof(VAAPIEncodePicture *));
> > +if (!ctx->encode_fifo)
> > +return AVERROR(ENOMEM);
> > +}
> > +#endif
> > +
> >  return 0;
> >
> >  fail:
> > @@ -2592,6 +2633,7 @@ av_cold int
> ff_vaapi_encode_close(AVCodecContext *avctx)
> >
> >  av_freep(&ctx->codec_sequence_params);
> >  av_freep(&ctx->codec_picture_params);
> > +av_fifo_freep(&ctx->encode_fifo);
> >
> >  av_buffer_unref(&ctx->recon_frames_ref);
> >  av_buffer_unref(&ctx->input_frames_ref);
> > diff --git a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h
> > index b41604a883..560a1c42a9 100644
> > --- a/libavcodec/vaapi_encode.h
> > +++ b/libavcodec/vaapi_encode.h
> > @@ -29,6 +29,7 @@
> >
> >  #include "libavutil/hwcontext.h"
> >  #include "libavutil/hwcontext_vaapi.h"
> > +#include "libavutil/fifo.h"
> >
> >  #include

Re: [FFmpeg-devel] [PATCH 3/3] libavcodec/qsvenc: Add intra refresh to hevc_qsv and add new intra refresh parameter

[Chen, Wenbin]

> On Mon, 2022-01-24 at 10:59 +0800, Wenbin Chen wrote:
> > Add intra refresh support to hevc_qsv as well.
> > Add an new intra refresh type: "horizontal", and an new param
> > ref_cycle_dist. This param specify the distance between the
> > beginnings of the intra-refresh cycles in frames.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi| 26 +-
> >  libavcodec/qsvenc.c  | 23 ++-
> >  libavcodec/qsvenc.h  |  1 +
> >  libavcodec/qsvenc_h264.c |  7 +--
> >  libavcodec/qsvenc_hevc.c |  9 +
> >  5 files changed, 54 insertions(+), 12 deletions(-)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 6c1c4df57a..4e35e50e4d 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -3344,7 +3344,8 @@ Specifies intra refresh type. The major goal of
> intra
> > refresh is improvement of
> >  error resilience without significant impact on encoded bitstream size
> caused
> > by
> >  I frames. The SDK encoder achieves this by encoding part of each frame in
> >  refresh cycle using intra MBs. @var{none} means no refresh. @var{vertical}
> > means
> > -vertical refresh, by column of MBs.
> > +vertical refresh, by column of MBs. To enable intra refresh, B frame
> should
> > be
> > +set to 0.
> >
> >  @item @var{int_ref_cycle_size}
> >  Specifies number of pictures within refresh cycle starting from 2. 0 and 1
> > are
> > @@ -3355,6 +3356,9 @@ Specifies QP difference for inserted intra MBs.
> This is
> > signed value in
> >  [-51, 51] range if target encoding bit-depth for luma samples is 8 and this
> >  range is [-63, 63] for 10 bit-depth or [-75, 75] for 12 bit-depth
> > respectively.
> >
> > +@item @var{int_ref_cycle_dist}
> > +Distance between the beginnings of the intra-refresh cycles in frames.
> > +
> >  @item @var{profile}
> >  @table @samp
> >  @item unknown
> > @@ -3463,6 +3467,26 @@ Insert picture timing SEI with pic_struct_syntax
> > element.
> >  @item @var{transform_skip}
> >  Turn this option ON to enable transformskip. It is supported on platform
> > equal
> >  or newer than ICL.
> > +
> > +@item @var{int_ref_type}
> > +Specifies intra refresh type. The major goal of intra refresh is
> improvement
> > of
> > +error resilience without significant impact on encoded bitstream size
> caused
> > by
> > +I frames. The SDK encoder achieves this by encoding part of each frame in
> > +refresh cycle using intra MBs. @var{none} means no refresh.
> @var{vertical}
> > means
> > +vertical refresh, by column of MBs. To enable intra refresh, B frame
> should
> > be
> > +set to 0.
> > +
> > +@item @var{int_ref_cycle_size}
> > +Specifies number of pictures within refresh cycle starting from 2. 0 and 1
> > are
> > +invalid values.
> > +
> > +@item @var{int_ref_qp_delta}
> > +Specifies QP difference for inserted intra MBs. This is signed value in
> > +[-51, 51] range if target encoding bit-depth for luma samples is 8 and this
> > +range is [-63, 63] for 10 bit-depth or [-75, 75] for 12 bit-depth
> > respectively.
> > +
> > +@item @var{int_ref_cycle_dist}
> > +Distance between the beginnings of the intra-refresh cycles in frames.
> >  @end table
> >
> >  @subsection MPEG2 Options
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index a8d876d6d9..af1529936e 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -267,8 +267,10 @@ static void dump_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q,
> >
> >  #if QSV_HAVE_CO2
> >  av_log(avctx, AV_LOG_VERBOSE,
> > -   "RecoveryPointSEI: %s IntRefType: %"PRIu16"; IntRefCycleSize:
> > %"PRIu16"; IntRefQPDelta: %"PRId16"\n",
> > -   print_threestate(co->RecoveryPointSEI), co2->IntRefType, co2-
> > >IntRefCycleSize, co2->IntRefQPDelta);
> > +   "RecoveryPointSEI: %s IntRefType: %"PRIu16"; IntRefCycleSize:
> > %"PRIu16
> > +   "; IntRefQPDelta: %"PRId16"; IntRefCycleDist: %"PRId16"\n",
> > +   print_threestate(co->RecoveryPointSEI), co2->IntRefType, co2-
> > >IntRefCycleSize,
> > +   co2->IntRefQPDelta, co3->IntRefCycleDist);
> 
> 
> IntRefCycleDist is a member of mfxExtCodingOption3, and was introduced in
> mfx
> version 1.16, so it should be used under QSV_HAVE_CO3 &&
> QSV_VERSION_ATLEAST(1,
> 16), not QSV_HAVE_CO2.
> 
> Thanks
> Haihao

Thanks for review. I will fix this.

Thanks
Wenbin

> 
> 
> >
> >  av_log(avctx, AV_LOG_VERBOSE, "MaxFrameSize: %d; ", co2-
> >MaxFrameSize);
> >  #if QSV_HAVE_MAX_SLICE_SIZE
> > @@ -865,13 +867,6 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >
> >  #if QSV_HAVE_CO2
> >  if (avctx->codec_id == AV_CODEC_ID_H264) {
> > -if (q->int_ref_type >= 0)
> > -q->extco2.IntRefType = q->int_ref_type;
> > -if (q->int_ref_cycle_size >= 0)
> > -q->extco2.IntRefCycleSize = q->int_ref_cycle_size;
> > -if (q->int_ref_qp_delta != INT16_MIN)
> > -  

Re: [FFmpeg-devel] [PATCH V5 1/2] libavcodec/vaapi_decode: fix the problem that init_pool_size < nb_surface

[Chen, Wenbin]

> For vaapi if the init_pool_size is not zero, the pool size is fixed.
> This means max surfaces is init_pool_size, but when mapping vaapi
> frame to qsv frame, the init_pool_size < nb_surface. The cause is that
> vaapi_decode_make_config() config the init_pool_size and it is called
> twice. The first time is to init frame_context and the second time is to
> init codec. On the second time the init_pool_size is changed to original
> value so the init_pool_size is lower than the reall size because
> pool_size used to initialize frame_context need to plus thread_count and
> 3 (guarantee 4 base work surfaces). Now add code to make sure
> init_pool_size is only set once. Now the following commandline works:
> 
> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> -hwaccel_output_format vaapi -i input.264 \
> -vf "hwmap=derive_device=qsv,format=qsv" \
> -c:v h264_qsv output.264
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/vaapi_decode.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavcodec/vaapi_decode.c b/libavcodec/vaapi_decode.c
> index 665af370ed..da0e72b10b 100644
> --- a/libavcodec/vaapi_decode.c
> +++ b/libavcodec/vaapi_decode.c
> @@ -650,7 +650,7 @@ int ff_vaapi_decode_init(AVCodecContext *avctx)
>  ctx->hwctx  = ctx->device->hwctx;
> 
>  err = vaapi_decode_make_config(avctx, ctx->frames->device_ref,
> -   &ctx->va_config, avctx->hw_frames_ctx);
> +   &ctx->va_config, NULL);
>  if (err)
>  goto fail;
> 
> --
> 2.25.1
> 

ping

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V5 2/2] libavutil/hwcontext_opencl: fix a bug for mapping qsv frame to opencl

[Chen, Wenbin]

> From: nyanmisaka 
> 
> mfxHDLPair was added to qsv, so modify qsv->opencl map function as well.
> Now the following commandline works:
> 
> ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 \
> -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -
> filter_hw_device ocl \
> -hwaccel qsv -hwaccel_output_format qsv -hwaccel_device qs -c:v h264_qsv
> \
> -i input.264 -vf
> "hwmap=derive_device=opencl,format=opencl,avgblur_opencl, \
> hwmap=derive_device=qsv:reverse=1:extra_hw_frames=32,format=qsv" \
> -c:v h264_qsv output.264
> 
> Signed-off-by: nyanmisaka 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_opencl.c | 14 +-
>  libavutil/hwcontext_qsv.c| 34 ++
>  2 files changed, 47 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_opencl.c b/libavutil/hwcontext_opencl.c
> index 26a3a24593..4e2ab18ede 100644
> --- a/libavutil/hwcontext_opencl.c
> +++ b/libavutil/hwcontext_opencl.c
> @@ -72,6 +72,12 @@
>  #include "hwcontext_drm.h"
>  #endif
> 
> +#if HAVE_OPENCL_VAAPI_INTEL_MEDIA && CONFIG_LIBMFX
> +extern int ff_qsv_get_surface_base_handle(mfxFrameSurface1 *surf,
> +  enum AVHWDeviceType base_dev_typ,
> +  void **base_handle);
> +#endif
> +
> 
>  typedef struct OpenCLDeviceContext {
>  // Default command queue to use for transfer/mapping operations on
> @@ -2248,8 +2254,14 @@ static int
> opencl_map_from_qsv(AVHWFramesContext *dst_fc, AVFrame *dst,
> 
>  #if CONFIG_LIBMFX
>  if (src->format == AV_PIX_FMT_QSV) {
> +void *base_handle;
>  mfxFrameSurface1 *mfx_surface = (mfxFrameSurface1*)src->data[3];
> -va_surface = *(VASurfaceID*)mfx_surface->Data.MemId;
> +err = ff_qsv_get_surface_base_handle(mfx_surface,
> + AV_HWDEVICE_TYPE_VAAPI,
> + &base_handle);
> +if (err < 0)
> +return err;
> +va_surface = *(VASurfaceID *)base_handle;
>  } else
>  #endif
>  if (src->format == AV_PIX_FMT_VAAPI) {
> diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> index 853fb7f60d..6d9b8324c2 100644
> --- a/libavutil/hwcontext_qsv.c
> +++ b/libavutil/hwcontext_qsv.c
> @@ -112,6 +112,40 @@ static const struct {
>  #endif
>  };
> 
> +extern int ff_qsv_get_surface_base_handle(mfxFrameSurface1 *surf,
> +  enum AVHWDeviceType base_dev_type,
> +  void **base_handle);
> +
> +/**
> + * Caller needs to allocate enough space for base_handle pointer.
> + **/
> +int ff_qsv_get_surface_base_handle(mfxFrameSurface1 *surf,
> +   enum AVHWDeviceType base_dev_type,
> +   void **base_handle)
> +{
> +mfxHDLPair *handle_pair;
> +handle_pair = surf->Data.MemId;
> +switch (base_dev_type) {
> +#if CONFIG_VAAPI
> +case AV_HWDEVICE_TYPE_VAAPI:
> +base_handle[0] = handle_pair->first;
> +return 0;
> +#endif
> +#if CONFIG_D3D11VA
> +case AV_HWDEVICE_TYPE_D3D11VA:
> +base_handle[0] = handle_pair->first;
> +base_handle[1] = handle_pair->secode;
> +return 0;
> +#endif
> +#if CONFIG_DXVA2
> +case AV_HWDEVICE_TYPE_DXVA2:
> +base_handle[0] = handle_pair->first;
> +return 0;
> +#endif
> +}
> +return AVERROR(EINVAL);
> +}
> +
>  static uint32_t qsv_fourcc_from_pix_fmt(enum AVPixelFormat pix_fmt)
>  {
>  int i;
> --
> 2.25.1
> 

ping

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V3 4/4] libavcodec/qsvenc: Add transform skip to hevc_qsv

[Chen, Wenbin]

> On Thu, 2022-01-13 at 13:12 +0800, Wenbin Chen wrote:
> > Add transform_skip option to hevc_qsv. By enabling this option,
> > the transform_skip_enabled_flag in PPS will be set to 1.
> > This option is supported on the platform equal or newer than CNL.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi|  4 
> >  libavcodec/qsvenc.c  | 10 +-
> >  libavcodec/qsvenc.h  |  1 +
> >  libavcodec/qsvenc_hevc.c |  3 +++
> >  4 files changed, 17 insertions(+), 1 deletion(-)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 3d7c944fba..7714084864 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -3447,6 +3447,10 @@ Number of rows for tiled encoding.
> >
> >  @item @var{aud}
> >  Insert the Access Unit Delimiter NAL.
> > +
> > +@item @var{transform_skip}
> > +Turn this option ON to enable transformskip. It is supported on platform
> > equal
> > +or newer than CNL.
> >  @end table
> >
> >  @subsection MPEG2 Options
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index f2ba0241c6..4e5997c9ed 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -358,6 +358,9 @@ static void dump_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q,
> >  av_log(avctx, AV_LOG_VERBOSE, "DisableDeblockingIdc: %"PRIu32" \n",
> co2-
> > >DisableDeblockingIdc);
> >  #endif
> >
> > +#if QSV_VERSION_ATLEAST(1, 26)
> > +av_log(avctx, AV_LOG_VERBOSE, "TransformSkip: %s \n",
> > print_threestate(co3->TransformSkip));
> > +#endif
> >  }
> >
> >  static void dump_video_vp9_param(AVCodecContext *avctx,
> QSVEncContext *q,
> > @@ -970,10 +973,15 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  #endif
> >  }
> >
> > +if (avctx->codec_id == AV_CODEC_ID_HEVC) {
> > +#if QSV_VERSION_ATLEAST(1, 26)
> > +q->extco3.TransformSkip = q->transform_skip ?
> MFX_CODINGOPTION_ON
> > :
> > +  
> > MFX_CODINGOPTION_OF
> > F;
> 
> q->extco3.TransformSkip is set to MFX_CODINGOPTION_ON when q-
> >transform_skip is
> -1, right ? TransformSkip is tri-state flag, we may set it to
> MFX_CODINGOPTION_UNKNOWN when q->transform_skip is -1.
> 
> Thanks
> Haihao

I will fix this in the next submission

Thanks
Wenbin
> 
> 
> > +#endif
> >  #if QSV_HAVE_GPB
> > -if (avctx->codec_id == AV_CODEC_ID_HEVC)
> >  q->extco3.GPB  = q->gpb ? MFX_CODINGOPTION_ON :
> > MFX_CODINGOPTION_OFF;
> >  #endif
> > +}
> >  q->extparam_internal[q->nb_extparam_internal++] = (mfxExtBuffer
> *)&q-
> > >extco3;
> >  #endif
> >  }
> > diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h
> > index 960604cb9a..5a574ada30 100644
> > --- a/libavcodec/qsvenc.h
> > +++ b/libavcodec/qsvenc.h
> > @@ -200,6 +200,7 @@ typedef struct QSVEncContext {
> >  int repeat_pps;
> >  int low_power;
> >  int gpb;
> > +int transform_skip;
> >
> >  int a53_cc;
> >
> > diff --git a/libavcodec/qsvenc_hevc.c b/libavcodec/qsvenc_hevc.c
> > index 08aba3011d..679c09c858 100644
> > --- a/libavcodec/qsvenc_hevc.c
> > +++ b/libavcodec/qsvenc_hevc.c
> > @@ -251,6 +251,9 @@ static const AVOption options[] = {
> >  { "tile_rows",  "Number of rows for tiled
> > encoding",  OFFSET(qsv.tile_rows),AV_OPT_TYPE_INT, { .i64 = 0 }, 0,
> > UINT16_MAX, VE },
> >  { "recovery_point_sei", "Insert recovery point SEI
> > messages",   OFFSET(qsv.recovery_point_sei),  AV_OPT_TYPE_INT,
> { .i64
> > = -1 },   -1,  1, VE },
> >  { "aud", "Insert the Access Unit Delimiter NAL", OFFSET(qsv.aud),
> > AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE},
> > +#if QSV_VERSION_ATLEAST(1, 26)
> > +{ "transform_skip", "Turn this option ON to enable
> > transformskip",   OFFSET(qsv.transform_skip),  AV_OPT_TYPE_INT,{
> > .i64 = -1},   -1, 1,  VE},
> > +#endif
> >
> >  { NULL },
> >  };
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V3 2/4] libavcodec/qsvenc: Add DisableDeblockingIdc support to qsv

[Chen, Wenbin]

> On Thu, 2022-01-13 at 13:12 +0800, Wenbin Chen wrote:
> > Add dblk_idc option to 264_qsv and hevc_qsv. Turining on this opion can
> > disable deblocking.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi   | 6 ++
> >  libavcodec/qsvenc.c | 8 
> >  libavcodec/qsvenc.h | 3 +++
> >  3 files changed, 17 insertions(+)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 68921fbd40..6fc94daa11 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -3299,6 +3299,9 @@ This flag controls changing of frame type from B
> to P.
> >  @item @var{b_strategy}
> >  This option controls usage of B frames as reference.
> >
> > +@item @var{dblk_idc}
> > +This option disable deblocking. It has value in range 0~2.
> > +
> >  @item @var{cavlc}
> >  If set, CAVLC is used; if unset, CABAC is used for encoding.
> >
> > @@ -3391,6 +3394,9 @@ Enable rate distortion optimization.
> >  @item @var{max_frame_size}
> >  Maximum encoded frame size in bytes.
> >
> > +@item @var{dblk_idc}
> > +This option disable deblocking. It has value in range 0~2.
> > +
> >  @item @var{idr_interval}
> >  Distance (in I-frames) between IDR frames.
> >  @table @samp
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index b3728d28d5..e87b69369a 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -346,6 +346,10 @@ static void dump_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q,
> >  av_log(avctx, AV_LOG_VERBOSE, "FrameRateExtD: %"PRIu32";
> FrameRateExtN:
> > %"PRIu32" \n",
> > info->FrameInfo.FrameRateExtD, info->FrameInfo.FrameRateExtN);
> >
> > +#if QSV_HAVE_DISABLEDEBLOCKIDC
> > +av_log(avctx, AV_LOG_VERBOSE, "DisableDeblockingIdc: %"PRIu32" \n",
> co2-
> > >DisableDeblockingIdc);
> > +#endif
> > +
> >  }
> >
> >  static void dump_video_vp9_param(AVCodecContext *avctx,
> QSVEncContext *q,
> > @@ -889,6 +893,10 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  q->extco2.ExtBRC = q->extbrc ? MFX_CODINGOPTION_ON :
> > MFX_CODINGOPTION_OFF;
> >  if (q->max_frame_size >= 0)
> >  q->extco2.MaxFrameSize = q->max_frame_size;
> > +#if QSV_HAVE_DISABLEDEBLOCKIDC
> > +if(q->dblk_idc >= 0)
> > +q->extco2.DisableDeblockingIdc = q->dblk_idc;
> > +#endif
> >
> >  #if QSV_VERSION_ATLEAST(1, 9)
> >  if (avctx->qmin >= 0 && avctx->qmax >= 0 && avctx->qmin > 
> > avctx-
> > >qmax) {
> > diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h
> > index 31516b8e55..aa49b35f07 100644
> > --- a/libavcodec/qsvenc.h
> > +++ b/libavcodec/qsvenc.h
> > @@ -44,6 +44,7 @@
> >
> >  #define QSV_HAVE_TRELLIS QSV_VERSION_ATLEAST(1, 8)
> >  #define QSV_HAVE_MAX_SLICE_SIZE QSV_VERSION_ATLEAST(1, 9)
> > +#define QSV_HAVE_DISABLEDEBLOCKIDC QSV_VERSION_ATLEAST(1, 9)
> >  #define QSV_HAVE_BREF_TYPE  QSV_VERSION_ATLEAST(1, 8)
> >
> >  #define QSV_HAVE_LA QSV_VERSION_ATLEAST(1, 7)
> > @@ -97,6 +98,7 @@
> >  { "b_strategy", "Strategy to choose between I/P/B-frames",
> > OFFSET(qsv.b_strategy),AV_OPT_TYPE_INT, { .i64 = -1 }, -1,  1, 
> > VE
> > }, \
> >  { "forced_idr", "Forcing I frames as IDR
> > frames", OFFSET(qsv.forced_idr), AV_OPT_TYPE_BOOL,{ .i64 =
> > 0  },  0,  1, VE }, \
> >  { "low_power", "enable low power mode(experimental: many limitations
> by mfx
> > version, BRC modes, etc.)", OFFSET(qsv.low_power), AV_OPT_TYPE_BOOL,
> { .i64 =
> > -1}, -1, 1, VE},\
> > +{ "dblk_idc", "This option disable deblocking. It has value in range
> > 0~2.",   OFFSET(qsv.dblk_idc),   AV_OPT_TYPE_INT,{ .i64 = -1 },   -
> > 1,  2,  VE},\
> 
> 
> The range is 0~2 in the help string, however the minimal value is -1, which
> will
> confuse user. We may set both the default and minimal values to 0 here.
> 
> Thanks
> Haihao

Sorry, I miss your comment. I will fix this.

Thanks
Wenbin

> 
> >
> >  extern const AVCodecHWConfigInternal *const ff_qsv_enc_hw_configs[];
> >
> > @@ -169,6 +171,7 @@ typedef struct QSVEncContext {
> >  int rdo;
> >  int max_frame_size;
> >  int max_slice_size;
> > +int dblk_idc;
> >
> >  int tile_cols;
> >  int tile_rows;
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V2] libavcodec/qsvenc: Add DisableDeblockingIdc support to qsv

[Chen, Wenbin]

> On Wed, 2022-01-12 at 13:28 +0800, Wenbin Chen wrote:
> > Add dblk_idc option to 264_qsv and hevc_qsv. Turining on this opion can
> > disable deblocking.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  doc/encoders.texi   | 6 ++
> >  libavcodec/qsvenc.c | 8 
> >  libavcodec/qsvenc.h | 3 +++
> >  3 files changed, 17 insertions(+)
> >
> > diff --git a/doc/encoders.texi b/doc/encoders.texi
> > index 68921fbd40..6fc94daa11 100644
> > --- a/doc/encoders.texi
> > +++ b/doc/encoders.texi
> > @@ -3299,6 +3299,9 @@ This flag controls changing of frame type from B
> to P.
> >  @item @var{b_strategy}
> >  This option controls usage of B frames as reference.
> >
> > +@item @var{dblk_idc}
> > +This option disable deblocking. It has value in range 0~2.
> > +
> >  @item @var{cavlc}
> >  If set, CAVLC is used; if unset, CABAC is used for encoding.
> >
> > @@ -3391,6 +3394,9 @@ Enable rate distortion optimization.
> >  @item @var{max_frame_size}
> >  Maximum encoded frame size in bytes.
> >
> > +@item @var{dblk_idc}
> > +This option disable deblocking. It has value in range 0~2.
> > +
> >  @item @var{idr_interval}
> >  Distance (in I-frames) between IDR frames.
> >  @table @samp
> > diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
> > index a13718652e..3a51d00ca9 100644
> > --- a/libavcodec/qsvenc.c
> > +++ b/libavcodec/qsvenc.c
> > @@ -346,6 +346,10 @@ static void dump_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q,
> >  av_log(avctx, AV_LOG_VERBOSE, "FrameRateExtD: %"PRIu32";
> FrameRateExtN:
> > %"PRIu32" \n",
> > info->FrameInfo.FrameRateExtD, info->FrameInfo.FrameRateExtN);
> >
> > +#if QSV_HAVE_DISABLEDEBLOCKIDC
> > +av_log(avctx, AV_LOG_VERBOSE, "DisableDeblockingIdc: %"PRIu32" \n",
> co2-
> > >DisableDeblockingIdc);
> > +#endif
> > +
> >  }
> >
> >  static void dump_video_vp9_param(AVCodecContext *avctx,
> QSVEncContext *q,
> > @@ -885,6 +889,10 @@ static int init_video_param(AVCodecContext
> *avctx,
> > QSVEncContext *q)
> >  q->extco2.ExtBRC = q->extbrc ? MFX_CODINGOPTION_ON :
> > MFX_CODINGOPTION_OFF;
> >  if (q->max_frame_size >= 0)
> >  q->extco2.MaxFrameSize = q->max_frame_size;
> > +#if QSV_HAVE_DISABLEDEBLOCKIDC
> > +if(q->dblk_idc >= 0)
> > +q->extco2.DisableDeblockingIdc = q->dblk_idc;
> > +#endif
> >
> >  #if QSV_VERSION_ATLEAST(1, 9)
> >  if (avctx->qmin >= 0 && avctx->qmax >= 0 && avctx->qmin > 
> > avctx-
> > >qmax) {
> > diff --git a/libavcodec/qsvenc.h b/libavcodec/qsvenc.h
> > index 31516b8e55..aa49b35f07 100644
> > --- a/libavcodec/qsvenc.h
> > +++ b/libavcodec/qsvenc.h
> > @@ -44,6 +44,7 @@
> >
> >  #define QSV_HAVE_TRELLIS QSV_VERSION_ATLEAST(1, 8)
> >  #define QSV_HAVE_MAX_SLICE_SIZE QSV_VERSION_ATLEAST(1, 9)
> > +#define QSV_HAVE_DISABLEDEBLOCKIDC QSV_VERSION_ATLEAST(1, 9)
> >  #define QSV_HAVE_BREF_TYPE  QSV_VERSION_ATLEAST(1, 8)
> >
> >  #define QSV_HAVE_LA QSV_VERSION_ATLEAST(1, 7)
> > @@ -97,6 +98,7 @@
> >  { "b_strategy", "Strategy to choose between I/P/B-frames",
> > OFFSET(qsv.b_strategy),AV_OPT_TYPE_INT, { .i64 = -1 }, -1,  1, 
> > VE
> > }, \
> >  { "forced_idr", "Forcing I frames as IDR
> > frames", OFFSET(qsv.forced_idr), AV_OPT_TYPE_BOOL,{ .i64 =
> > 0  },  0,  1, VE }, \
> >  { "low_power", "enable low power mode(experimental: many limitations
> by mfx
> > version, BRC modes, etc.)", OFFSET(qsv.low_power), AV_OPT_TYPE_BOOL,
> { .i64 =
> > -1}, -1, 1, VE},\
> > +{ "dblk_idc", "This option disable deblocking. It has value in range
> > 0~2.",   OFFSET(qsv.dblk_idc),   AV_OPT_TYPE_INT,{ .i64 = -1 },   -
> > 1,  2,  VE},\
> >
> >  extern const AVCodecHWConfigInternal *const ff_qsv_enc_hw_configs[];
> >
> > @@ -169,6 +171,7 @@ typedef struct QSVEncContext {
> >  int rdo;
> >  int max_frame_size;
> >  int max_slice_size;
> > +int dblk_idc;
> >
> >  int tile_cols;
> >  int tile_rows;
> 
> 
> patchwork failed to apply this patch, see
> https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220112052850.1147
> 160-1-wenbin.c...@intel.com/
> 
> Thanks
> Haihao

I see. I should submit patches in one patchset. I will resubmit them.

Thanks
Wenbin
> 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V4 2/2] libavutil/hwcontext_opencl: fix a bug for mapping qsv frame to opencl

[Chen, Wenbin]

> On Fri, 2022-01-07 at 15:36 +0800, Wenbin Chen wrote:
> > From: nyanmisaka 
> >
> > mfxHDLPair was added to qsv, so modify qsv->opencl map function as well.
> > Now the following commandline works:
> >
> > ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 \
> > -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -
> filter_hw_device ocl
> > \
> > -hwaccel qsv -hwaccel_output_format qsv -hwaccel_device qs -c:v
> h264_qsv \
> > -i input.264 -vf
> "hwmap=derive_device=opencl,format=opencl,avgblur_opencl, \
> > hwmap=derive_device=qsv:reverse=1:extra_hw_frames=32,format=qsv" \
> > -c:v h264_qsv output.264
> >
> > Signed-off-by: nyanmisaka 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_opencl.c | 14 +-
> >  libavutil/hwcontext_qsv.c| 34 ++
> >  2 files changed, 47 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavutil/hwcontext_opencl.c b/libavutil/hwcontext_opencl.c
> > index 26a3a24593..ceaef26f1c 100644
> > --- a/libavutil/hwcontext_opencl.c
> > +++ b/libavutil/hwcontext_opencl.c
> > @@ -72,6 +72,12 @@
> >  #include "hwcontext_drm.h"
> >  #endif
> >
> > +#if HAVE_OPENCL_VAAPI_INTEL_MEDIA
> > +extern int ff_qsv_get_surface_base_handle(mfxFrameSurface1 *surf,
> > +  enum AVHWDeviceType base_dev_typ,
> > +  void **base_handle);
> > +#endif
> 
> opencl_vaapi_intel_media can be enabled without libmfx.
> 
> Thanks
> Haihao
> 
Thanks for review.
I will update.

Thanks
Wenbin
> 
> > +
> >
> >  typedef struct OpenCLDeviceContext {
> >  // Default command queue to use for transfer/mapping operations on
> > @@ -2248,8 +2254,14 @@ static int
> opencl_map_from_qsv(AVHWFramesContext
> > *dst_fc, AVFrame *dst,
> >
> >  #if CONFIG_LIBMFX
> >  if (src->format == AV_PIX_FMT_QSV) {
> > +void *base_handle;
> >  mfxFrameSurface1 *mfx_surface = (mfxFrameSurface1*)src->data[3];
> > -va_surface = *(VASurfaceID*)mfx_surface->Data.MemId;
> > +err = ff_qsv_get_surface_base_handle(mfx_surface,
> > + AV_HWDEVICE_TYPE_VAAPI,
> > + &base_handle);
> > +if (err < 0)
> > +return err;
> > +va_surface = *(VASurfaceID *)base_handle;
> >  } else
> >  #endif
> >  if (src->format == AV_PIX_FMT_VAAPI) {
> > diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> > index 853fb7f60d..6d9b8324c2 100644
> > --- a/libavutil/hwcontext_qsv.c
> > +++ b/libavutil/hwcontext_qsv.c
> > @@ -112,6 +112,40 @@ static const struct {
> >  #endif
> >  };
> >
> > +extern int ff_qsv_get_surface_base_handle(mfxFrameSurface1 *surf,
> > +  enum AVHWDeviceType 
> > base_dev_type,
> > +  void **base_handle);
> > +
> > +/**
> > + * Caller needs to allocate enough space for base_handle pointer.
> > + **/
> > +int ff_qsv_get_surface_base_handle(mfxFrameSurface1 *surf,
> > +   enum AVHWDeviceType base_dev_type,
> > +   void **base_handle)
> > +{
> > +mfxHDLPair *handle_pair;
> > +handle_pair = surf->Data.MemId;
> > +switch (base_dev_type) {
> > +#if CONFIG_VAAPI
> > +case AV_HWDEVICE_TYPE_VAAPI:
> > +base_handle[0] = handle_pair->first;
> > +return 0;
> > +#endif
> > +#if CONFIG_D3D11VA
> > +case AV_HWDEVICE_TYPE_D3D11VA:
> > +base_handle[0] = handle_pair->first;
> > +base_handle[1] = handle_pair->secode;
> > +return 0;
> > +#endif
> > +#if CONFIG_DXVA2
> > +case AV_HWDEVICE_TYPE_DXVA2:
> > +base_handle[0] = handle_pair->first;
> > +return 0;
> > +#endif
> > +}
> > +return AVERROR(EINVAL);
> > +}
> > +
> >  static uint32_t qsv_fourcc_from_pix_fmt(enum AVPixelFormat pix_fmt)
> >  {
> >  int i;
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V2 2/3] libavcodec/vaapi_encode: Change the way to call async to increase performance

[Chen, Wenbin]

> Wenbin Chen:
> > Fix: #7706. After commit 5fdcf85bbffe7451c2, vaapi encoder's performance
> > decrease. The reason is that vaRenderPicture() and vaSyncBuffer() are
> > called at the same time (vaRenderPicture() always followed by a
> > vaSyncBuffer()). When we encode stream with B frames, we need buffer to
> > reorder frames, so we can send serveral frames to HW at once to increase
> > performance. Now I changed them to be called in a asynchronous way,
> which
> > will make better use of hardware. 1080p transcoding increases about 17%
> > fps on my environment.
> >
> > This change fits vaSyncBuffer(), so if driver does not support
> > vaSyncBuffer, it will keep previous operation.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/vaapi_encode.c | 64 -
> --
> >  libavcodec/vaapi_encode.h |  5 +++
> >  2 files changed, 58 insertions(+), 11 deletions(-)
> >
> > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> > index b87b58a42b..9a3b3ba4ad 100644
> > --- a/libavcodec/vaapi_encode.c
> > +++ b/libavcodec/vaapi_encode.c
> > @@ -984,8 +984,10 @@ static int
> vaapi_encode_pick_next(AVCodecContext *avctx,
> >  if (!pic && ctx->end_of_stream) {
> >  --b_counter;
> >  pic = ctx->pic_end;
> > -if (pic->encode_issued)
> > +if (pic->encode_complete)
> >  return AVERROR_EOF;
> > +else if (pic->encode_issued)
> > +return AVERROR(EAGAIN);
> >  }
> >
> >  if (!pic) {
> > @@ -1210,18 +1212,45 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
> >  return AVERROR(EAGAIN);
> >  }
> >
> > -pic = NULL;
> > -err = vaapi_encode_pick_next(avctx, &pic);
> > -if (err < 0)
> > -return err;
> > -av_assert0(pic);
> > +#if VA_CHECK_VERSION(1, 9, 0)
> > +if (ctx->has_sync_buffer_func) {
> > +while (av_fifo_size(ctx->encode_fifo) <=
> > +   MAX_PICTURE_REFERENCES * sizeof(VAAPIEncodePicture *)) {
> > +pic = NULL;
> > +err = vaapi_encode_pick_next(avctx, &pic);
> > +if (err < 0)
> > +break;
> > +
> > +av_assert0(pic);
> > +pic->encode_order = ctx->encode_order +
> > +(av_fifo_size(ctx->encode_fifo) / 
> > sizeof(VAAPIEncodePicture *));
> > +err = vaapi_encode_issue(avctx, pic);
> > +if (err < 0) {
> > +av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > +return err;
> > +}
> > +av_fifo_generic_write(ctx->encode_fifo, &pic, sizeof(pic), 
> > NULL);
> > +}
> > +if (!av_fifo_size(ctx->encode_fifo))
> > +return err;
> > +av_fifo_generic_read(ctx->encode_fifo, &pic, sizeof(pic), NULL);
> > +ctx->encode_order = pic->encode_order + 1;
> > +} else
> > +#endif
> > +{
> > +pic = NULL;
> > +err = vaapi_encode_pick_next(avctx, &pic);
> > +if (err < 0)
> > +return err;
> > +av_assert0(pic);
> >
> > -pic->encode_order = ctx->encode_order++;
> > +pic->encode_order = ctx->encode_order++;
> >
> > -err = vaapi_encode_issue(avctx, pic);
> > -if (err < 0) {
> > -av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > -return err;
> > +err = vaapi_encode_issue(avctx, pic);
> > +if (err < 0) {
> > +av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > +return err;
> > +}
> >  }
> >
> >  err = vaapi_encode_output(avctx, pic, pkt);
> > @@ -2555,6 +2584,18 @@ av_cold int
> ff_vaapi_encode_init(AVCodecContext *avctx)
> >  }
> >  }
> >
> > +#if VA_CHECK_VERSION(1, 9, 0)
> > +//check vaSyncBuffer function
> > +vas = vaSyncBuffer(ctx->hwctx->display, 0, 0);
> > +if (vas != VA_STATUS_ERROR_UNIMPLEMENTED) {
> > +ctx->has_sync_buffer_func = 1;
> > +ctx->encode_fifo = av_fifo_alloc((MAX_PICTURE_REFERENCES + 1) *
> > +sizeof(VAAPIEncodePicture *));
> > +if (!ctx->encode_fifo)
> > +return AVERROR(ENOMEM);
> > +}
> > +#endif
> > +
> >  return 0;
> >
> >  fail:
> > @@ -2592,6 +2633,7 @@ av_cold int
> ff_vaapi_encode_close(AVCodecContext *avctx)
> >
> >  av_freep(&ctx->codec_sequence_params);
> >  av_freep(&ctx->codec_picture_params);
> > +av_fifo_freep(&ctx->encode_fifo);
> 
> Is it guaranteed that the fifo is empty at this point? I don't think so.

I don't check the fifo size, because in ff_vaapi_encode_close() all pics
are already freed and encode_fifo only buffer pic.
```
for (pic = ctx->pic_start; pic; pic = next) {
next = pic->next;
vaapi_encode_free(avctx, pic);
}
```

> 
> >
> >  av_buffer_unref(&ctx->recon_frames_ref);
> >  av_buffer_unref(&ctx->input_frames_ref);
> > diff --git a/libavcodec/vaapi_encode.h b/liba

Re: [FFmpeg-devel] [FFmpeg-cvslog] libavutil/hwcontext_vaapi: Add a new nv12 format map to support vulkan frame

[Chen, Wenbin]

> On 10/12/2021 16:05, Wenbin Chen wrote:
> > ffmpeg | branch: master | Wenbin Chen  | Tue
> Dec  7 17:05:50 2021 +0800| [f3c9847c2754b7a43eb721c95e356a53085c2491]
> | committer: Lynne
> >
> > libavutil/hwcontext_vaapi: Add a new nv12 format map to support vulkan
> frame
> >
> > Vulkan will map nv12 to R8 and GR88, so add this map to vaapi to support
> > vulkan frame.
> >
> > Signed-off-by: Wenbin Chen 
> >
> >>
> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=f3c9847c2754b7a
> 43eb721c95e356a53085c2491
> > ---
> >
> >   libavutil/hwcontext_vaapi.c | 1 +
> >   1 file changed, 1 insertion(+)
> >
> > diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> > index 75acc851d6..994b744e4d 100644
> > --- a/libavutil/hwcontext_vaapi.c
> > +++ b/libavutil/hwcontext_vaapi.c
> > @@ -992,6 +992,7 @@ static const struct {
> >   } vaapi_drm_format_map[] = {
> >   #ifdef DRM_FORMAT_R8
> >   DRM_MAP(NV12, 2, DRM_FORMAT_R8,  DRM_FORMAT_RG88),
> > +DRM_MAP(NV12, 2, DRM_FORMAT_R8,  DRM_FORMAT_GR88),
> >   #endif
> >   DRM_MAP(NV12, 1, DRM_FORMAT_NV12),
> >   #if defined(VA_FOURCC_P010) && defined(DRM_FORMAT_R16)
> 
> This looks very shady.  Shouldn't one or the other of these be NV21, with the
> second plane VU rather than UV?
> 
> - Mark

I add this because I see vulkan map RG88 and GR88 to the same format.
```
{ DRM_FORMAT_GR88, VK_FORMAT_R8G8_UNORM },
{ DRM_FORMAT_RG88, VK_FORMAT_R8G8_UNORM },
```
I thinks you are right. One of them should be NV21. I should switch the position
Of GR88 and RG88 in this map table so that VK_FORMAT_R8G8_UNORM can be
mapped to DRM_FORMAT_RG88 rather than DRM_FORMAT_GR88.

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] libavcodec/vaapi_encode: Change the way to call async to increase performance

[Chen, Wenbin]

> On 27/10/2021 09:57, Wenbin Chen wrote:
> > Fix: #7706. After commit 5fdcf85bbffe7451c2, vaapi encoder's performance
> > decrease. The reason is that vaRenderPicture() and vaSyncSurface() are
> > called at the same time (vaRenderPicture() always followed by a
> > vaSyncSurface()). When we encode stream with B frames, we need buffer
> to
> > reorder frames, so we can send serveral frames to HW at once to increase
> > performance. Now I changed them to be called in a
> > asynchronous way, which will make better use of hardware.
> > 1080p transcoding increases about 17% fps on my environment.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >   libavcodec/vaapi_encode.c | 41 ---
> >   libavcodec/vaapi_encode.h |  3 +++
> >   2 files changed, 33 insertions(+), 11 deletions(-)
> 
> The API does not allow this behaviour.
> 
> For some bizarre reason (I think a badly-written example combined with the
> Intel driver being synchronous in vaEndPicture() for a long time), the sync to
> a surface is to the /input/ surface of an encode rather than the output
> surface.
> 
> That means you can't have multiple encodes outstanding on the same
> surface and expect to sync usefully, because the only argument to
> vaSyncSurface() is the surface to sync to without anything about the
> associated context.
> 
> Therefore trying to make it asynchronous like this falls down when input
> surfaces might appear multiple times, or might be used in the input of
> multiple encoders, because you can't tell whether your sync means the thing
> you actually wanted to finish has finished.
> 
> (The commit you point to above as having decreased performance fixed this
> bug, since it became much more visible with decoupled send/receive.)
> 
> So: put this change after the switch to syncing on output buffers (since that
> operation does make sense for this), and leave the existing behaviour for
> cases where you have to sync on the input surface.
> 
> - Mark

Thanks for your advice. It makes sense to me. I will update the patches

Best Regards
Wenbin
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] libavcodec/vaapi_encode: Change the way to call async to increase performance

[Chen, Wenbin]

> On Wed, 2021-10-27 at 16:57 +0800, Wenbin Chen wrote:
> > Fix: #7706. After commit 5fdcf85bbffe7451c2, vaapi encoder's performance
> > decrease. The reason is that vaRenderPicture() and vaSyncSurface() are
> > called at the same time (vaRenderPicture() always followed by a
> > vaSyncSurface()). When we encode stream with B frames, we need buffer
> to
> > reorder frames, so we can send serveral frames to HW at once to increase
> > performance. Now I changed them to be called in a
> > asynchronous way, which will make better use of hardware.
> > 1080p transcoding increases about 17% fps on my environment.
> 
> Could you provide your command ? I'd like to have a try.
> 
> Thanks
> Haihao

Here is my command:
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i input.264 -c:v h264_vaapi 
output.264

Thanks
Wenbin

> 
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/vaapi_encode.c | 41 ---
> >  libavcodec/vaapi_encode.h |  3 +++
> >  2 files changed, 33 insertions(+), 11 deletions(-)
> >
> > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> > index ec054ae701..5927849233 100644
> > --- a/libavcodec/vaapi_encode.c
> > +++ b/libavcodec/vaapi_encode.c
> > @@ -951,8 +951,10 @@ static int
> vaapi_encode_pick_next(AVCodecContext *avctx,
> >  if (!pic && ctx->end_of_stream) {
> >  --b_counter;
> >  pic = ctx->pic_end;
> > -if (pic->encode_issued)
> > +if (pic->encode_complete)
> >  return AVERROR_EOF;
> > +else if (pic->encode_issued)
> > +return AVERROR(EAGAIN);
> >  }
> >
> >  if (!pic) {
> > @@ -1177,20 +1179,31 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext
> > *avctx, AVPacket *pkt)
> >  return AVERROR(EAGAIN);
> >  }
> >
> > -pic = NULL;
> > -err = vaapi_encode_pick_next(avctx, &pic);
> > -if (err < 0)
> > -return err;
> > -av_assert0(pic);
> > +while (av_fifo_size(ctx->encode_fifo) <= MAX_PICTURE_REFERENCES *
> > sizeof(VAAPIEncodePicture *)) {
> > +pic = NULL;
> > +err = vaapi_encode_pick_next(avctx, &pic);
> > +if (err < 0)
> > +break;
> > +av_assert0(pic);
> >
> > -pic->encode_order = ctx->encode_order++;
> > +pic->encode_order = ctx->encode_order +
> > +(av_fifo_size(ctx->encode_fifo) /
> > sizeof(VAAPIEncodePicture *));
> >
> > -err = vaapi_encode_issue(avctx, pic);
> > -if (err < 0) {
> > -av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > -return err;
> > +err = vaapi_encode_issue(avctx, pic);
> > +if (err < 0) {
> > +av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> > +return err;
> > +}
> > +
> > +av_fifo_generic_write(ctx->encode_fifo, &pic, sizeof(pic), NULL);
> >  }
> >
> > +if (!av_fifo_size(ctx->encode_fifo))
> > +return err;
> > +
> > +av_fifo_generic_read(ctx->encode_fifo, &pic, sizeof(pic), NULL);
> > +ctx->encode_order = pic->encode_order + 1;
> > +
> >  err = vaapi_encode_output(avctx, pic, pkt);
> >  if (err < 0) {
> >  av_log(avctx, AV_LOG_ERROR, "Output failed: %d.\n", err);
> > @@ -2520,6 +2533,11 @@ av_cold int
> ff_vaapi_encode_init(AVCodecContext *avctx)
> >  }
> >  }
> >
> > +ctx->encode_fifo = av_fifo_alloc((MAX_PICTURE_REFERENCES + 1) *
> > +  sizeof(VAAPIEncodePicture *));
> > +if (!ctx->encode_fifo)
> > +return AVERROR(ENOMEM);
> > +
> >  return 0;
> >
> >  fail:
> > @@ -2552,6 +2570,7 @@ av_cold int
> ff_vaapi_encode_close(AVCodecContext *avctx)
> >
> >  av_freep(&ctx->codec_sequence_params);
> >  av_freep(&ctx->codec_picture_params);
> > +av_fifo_freep(&ctx->encode_fifo);
> >
> >  av_buffer_unref(&ctx->recon_frames_ref);
> >  av_buffer_unref(&ctx->input_frames_ref);
> > diff --git a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h
> > index b41604a883..89fe8de466 100644
> > --- a/libavcodec/vaapi_encode.h
> > +++ b/libavcodec/vaapi_encode.h
> > @@ -29,6 +29,7 @@
> >
> >  #include "libavutil/hwcontext.h"
> >  #include "libavutil/hwcontext_vaapi.h"
> > +#include "libavutil/fifo.h"
> >
> >  #include "avcodec.h"
> >  #include "hwconfig.h"
> > @@ -345,6 +346,8 @@ typedef struct VAAPIEncodeContext {
> >  int roi_warned;
> >
> >  AVFrame *frame;
> > +
> > +AVFifoBuffer *encode_fifo;
> >  } VAAPIEncodeContext;
> >
> >  enum {
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V3 2/2] libavutil/hwcontext_opencl: fix a bug for mapping qsv frame to opencl

[Chen, Wenbin]

> On Fri, 2021-12-10 at 13:38 +0800, Wenbin Chen wrote:
> > From: nyanmisaka 
> >
> > mfxHDLPair was added to qsv, so modify qsv->opencl map function as well.
> > Now the following commandline works:
> >
> > ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 \
> > -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -
> filter_hw_device ocl
> > \
> > -hwaccel qsv -hwaccel_output_format qsv -hwaccel_device qs -c:v
> h264_qsv \
> > -i input.264 -vf
> "hwmap=derive_device=opencl,format=opencl,avgblur_opencl, \
> > hwmap=derive_device=qsv:reverse=1:extra_hw_frames=32,format=qsv" \
> > -c:v h264_qsv output.264
> >
> > Signed-off-by: nyanmisaka 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_opencl.c | 3 ++-
> >  libavutil/hwcontext_qsv.h| 5 +
> >  2 files changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavutil/hwcontext_opencl.c b/libavutil/hwcontext_opencl.c
> > index 26a3a24593..ab812999cd 100644
> > --- a/libavutil/hwcontext_opencl.c
> > +++ b/libavutil/hwcontext_opencl.c
> > @@ -48,6 +48,7 @@
> >  #if HAVE_OPENCL_VAAPI_INTEL_MEDIA
> >  #if CONFIG_LIBMFX
> >  #include 
> > +#include "hwcontext_qsv.h"
> >  #endif
> >  #include 
> >  #include 
> > @@ -2249,7 +2250,7 @@ static int
> opencl_map_from_qsv(AVHWFramesContext
> > *dst_fc, AVFrame *dst,
> >  #if CONFIG_LIBMFX
> >  if (src->format == AV_PIX_FMT_QSV) {
> >  mfxFrameSurface1 *mfx_surface = (mfxFrameSurface1*)src->data[3];
> > -va_surface = *(VASurfaceID*)mfx_surface->Data.MemId;
> > +va_surface = *MFXSURFACEP_TO_VASURFACEP(mfx_surface);
> >  } else
> >  #endif
> >  if (src->format == AV_PIX_FMT_VAAPI) {
> > diff --git a/libavutil/hwcontext_qsv.h b/libavutil/hwcontext_qsv.h
> > index b98d611cfc..957df01ef1 100644
> > --- a/libavutil/hwcontext_qsv.h
> > +++ b/libavutil/hwcontext_qsv.h
> > @@ -29,6 +29,11 @@
> >   * contain AVBufferRefs whose data pointer points to an
> mfxFrameSurface1
> > struct.
> >   */
> >
> > +#if CONFIG_VAAPI
> > +#define MFXSURFACEP_TO_VASURFACEP(surf) \
> > +(VASurfaceID*)(((mfxHDLPair*)surf->Data.MemId)->first)
> > +#endif
> > +
> 
> Is it possible to remove the dependency on VAAPI in a public qsv header file ?
> 
> Thanks
> Haihao

How about moving this to qsv_internal.h?

> 
> >  /**
> >   * This struct is allocated as AVHWDeviceContext.hwctx
> >   */
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: clean padding when upload qsv frames

[Chen, Wenbin]

> On Thu, 2021-12-02 at 15:40 +0800, Wenbin Chen wrote:
> > When we upload a frame that is not padded as MSDK requires, we create a
> > new AVFrame to copy data. The frame's padding data is uninitialized so
> > it brings run to run problem. For example, If we run the following
> > command serveral times we will get different outputs.
> >
> > ffmpeg -init_hw_device qsv=qsv:hw -qsv_device /dev/dri/renderD128
> > -filter_hw_device qsv -f rawvideo -s 192x200 -pix_fmt p010
> > -i 192x200_P010.yuv -vf "format=nv12,hwupload=extra_hw_frames=16"
> > -c:v hevc_qsv output.265
> >
> > According to
> > https://github.com/Intel-Media-
> SDK/MediaSDK/blob/master/doc/mediasdk-man.md#encoding-procedures
> > "Note: It is the application's responsibility to fill pixels outside
> > of crop window when it is smaller than frame to be encoded. Especially
> > in cases when crops are not aligned to minimum coding block size (16
> > for AVC, 8 for HEVC and VP9)"
> >
> > I add a function to fill padding area with border pixel to fix this
> > run2run problem, and also move the new AVFrame to global structure
> > to reduce redundant allocation operation to increase preformance.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_qsv.c | 96 +-
> -
> >  1 file changed, 83 insertions(+), 13 deletions(-)
> >
> > diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> > index 268be9f8a1..983494666b 100644
> > --- a/libavutil/hwcontext_qsv.c
> > +++ b/libavutil/hwcontext_qsv.c
> > @@ -47,6 +47,7 @@
> >  #include "pixfmt.h"
> >  #include "pixdesc.h"
> >  #include "time.h"
> > +#include "imgutils.h"
> >
> >  #define QSV_VERSION_ATLEAST(MAJOR, MINOR)   \
> >  (MFX_VERSION_MAJOR > (MAJOR) || \
> > @@ -90,6 +91,7 @@ typedef struct QSVFramesContext {
> >
> >  mfxExtOpaqueSurfaceAlloc opaque_alloc;
> >  mfxExtBuffer *ext_buffers[1];
> > +AVFrame realigned_tmp_frame;
> >  } QSVFramesContext;
> >
> >  static const struct {
> > @@ -137,6 +139,54 @@ static uint32_t qsv_get_d3d11va_bind_flags(int
> mem_type)
> >  }
> >  #endif
> >
> > +static int qsv_fill_border(AVFrame *dst, const AVFrame *src)
> > +{
> > +const AVPixFmtDescriptor *desc;
> > +int i, planes_nb = 0;
> > +if (dst->format != src->format)
> > +return AVERROR(EINVAL);
> > +
> > +desc = av_pix_fmt_desc_get(dst->format);
> > +
> > +for (i = 0; i < desc->nb_components; i++)
> > +planes_nb = FFMAX(planes_nb, desc->comp[i].plane + 1);
> > +
> > +for (i = 0; i < planes_nb; i++) {
> > +int sheight, dheight, y;
> > +ptrdiff_t swidth = av_image_get_linesize(src->format,
> > + src->width,
> > + i);
> > +ptrdiff_t dwidth = av_image_get_linesize(dst->format,
> > + dst->width,
> > + i);
> > +const AVComponentDescriptor comp = desc->comp[i];
> > +if (swidth < 0 || dwidth < 0) {
> > +av_log(NULL, AV_LOG_ERROR, "av_image_get_linesize failed\n");
> > +return AVERROR(EINVAL);
> > +}
> > +sheight = src->height;
> > +dheight = dst->height;
> > +if (i) {
> > +sheight = AV_CEIL_RSHIFT(src->height, desc->log2_chroma_h);
> > +dheight = AV_CEIL_RSHIFT(dst->height, desc->log2_chroma_h);
> > +}
> > +//fill right padding
> > +for (y = 0; y < sheight; y++) {
> > +void *line_ptr = dst->data[i] + y*dst->linesize[i] + swidth;
> > +av_memcpy_backptr(line_ptr,
> > +   comp.depth > 8 ? 2 : 1,
> > +   dwidth - swidth);
> > +}
> > +//fill bottom padding
> > +for (y = sheight; y < dheight; y++) {
> > +memcpy(dst->data[i]+y*dst->linesize[i],
> > +   dst->data[i]+(sheight-1)*dst->linesize[i],
> > +   dwidth);
> > +}
> > +}
> > +return 0;
> > +}
> > +
> >  static int qsv_device_init(AVHWDeviceContext *ctx)
> >  {
> >  AVQSVDeviceContext *hwctx = ctx->hwctx;
> > @@ -220,6 +270,7 @@ static void
> qsv_frames_uninit(AVHWFramesContext *ctx)
> >  av_freep(&s->surface_ptrs);
> >  av_freep(&s->surfaces_internal);
> >  av_freep(&s->handle_pairs_internal);
> > +av_frame_unref(&s->realigned_tmp_frame);
> >  av_buffer_unref(&s->child_frames_ref);
> >  }
> >
> > @@ -1014,12 +1065,13 @@ static int
> qsv_transfer_data_to(AVHWFramesContext
> > *ctx, AVFrame *dst,
> >  QSVFramesContext   *s = ctx->internal->priv;
> >  mfxFrameSurface1   in = {{ 0 }};
> >  mfxFrameSurface1 *out = (mfxFrameSurface1*)dst->data[3];
> > +mfxFrameInfo tmp_info;
> >
> >  mfxSyncPoint sync = NULL;
> >  mfxStatus err;
> >  int ret = 0;
> >  /* make a copy if the input is not padded as libmfx requires */
> > 

Re: [FFmpeg-devel] [PATCH] libavfilter/vf_overlay_qsv: Use format of first input to set output format for overlay_qsv

[Chen, Wenbin]

> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> Wenbin Chen
> > Sent: Friday, December 10, 2021 3:22 AM
> > To: ffmpeg-devel@ffmpeg.org
> > Cc: Wenbin Chen 
> > Subject: [FFmpeg-devel] [PATCH] libavfilter/vf_overlay_qsv: Use format of
> > first input to set output format for overlay_qsv
> >
> > overlay_qsv hard coded to use nv12 as output format. Now use the format
> > of the first input to set output format. Now the following command
> > works:
> >
> > ffmpeg -hwaccel qsv -c:v hevc_qsv -i input_p010.265 -hwaccel qsv -c:v
> > hevc_qsv \
> > -i input2_p010.265 -filter_complex
> "[0:v][1:v]overlay_qsv=x=0:y=0:alpha=255,
> > \
> > hwdownload,format=p010le" -f rawvideo -y output_p010.yuv
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavfilter/vf_overlay_qsv.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/libavfilter/vf_overlay_qsv.c b/libavfilter/vf_overlay_qsv.c
> > index 7e76b39aa9..d947a1faa1 100644
> > --- a/libavfilter/vf_overlay_qsv.c
> > +++ b/libavfilter/vf_overlay_qsv.c
> > @@ -276,6 +276,7 @@ static int config_output(AVFilterLink *outlink)
> >  int ret;
> >
> >  av_log(ctx, AV_LOG_DEBUG, "Output is of %s.\n",
> > av_get_pix_fmt_name(outlink->format));
> > +vpp->qsv_param.out_sw_format = in0->format;
> >  if ((in0->format == AV_PIX_FMT_QSV && in1->format !=
> AV_PIX_FMT_QSV) ||
> >  (in0->format != AV_PIX_FMT_QSV && in1->format ==
> AV_PIX_FMT_QSV)) {
> >  av_log(ctx, AV_LOG_ERROR, "Mixing hardware and software pixel
> > formats is not supported.\n");
> > @@ -288,6 +289,7 @@ static int config_output(AVFilterLink *outlink)
> >  av_log(ctx, AV_LOG_ERROR, "Inputs with different underlying QSV
> > devices are forbidden.\n");
> >  return AVERROR(EINVAL);
> >  }
> > +vpp->qsv_param.out_sw_format = hw_frame0->sw_format;
> >  }
> >
> >  outlink->w  = vpp->var_values[VAR_MW];
> > --
> 
> When you're already looking at that area:
> 
> Do you have any idea why it sets the output time_base to the reciprocal
> input0 framerate?
> I think it should rather set it to the same as the input0 time_base.
> 
> https://github.com/FFmpeg/FFmpeg/blob/4f44a218e53cd92e64ba10a935bc
> 1e7583c3e218/libavfilter/vf_overlay_qsv.c#L293-L296
> 
> What do you think? (also @Haihao)
> 
> Thanks,
> softworkz

I have no idea why it is set to the reciprocal input0 framerate.
I agree with you. I think it should be set to in0->time_base.
I tried and changing to "in0->time_base" also works fine.
 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavfilter/vf_overlay_qsv: Use format of first input to set output format for overlay_qsv

[Chen, Wenbin]

> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> Wenbin Chen
> > Sent: Friday, December 10, 2021 3:22 AM
> > To: ffmpeg-devel@ffmpeg.org
> > Cc: Wenbin Chen 
> > Subject: [FFmpeg-devel] [PATCH] libavfilter/vf_overlay_qsv: Use format of
> > first input to set output format for overlay_qsv
> >
> > overlay_qsv hard coded to use nv12 as output format. Now use the format
> > of the first input to set output format. Now the following command
> > works:
> >
> > ffmpeg -hwaccel qsv -c:v hevc_qsv -i input_p010.265 -hwaccel qsv -c:v
> > hevc_qsv \
> > -i input2_p010.265 -filter_complex
> "[0:v][1:v]overlay_qsv=x=0:y=0:alpha=255,
> > \
> > hwdownload,format=p010le" -f rawvideo -y output_p010.yuv
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavfilter/vf_overlay_qsv.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/libavfilter/vf_overlay_qsv.c b/libavfilter/vf_overlay_qsv.c
> > index 7e76b39aa9..d947a1faa1 100644
> > --- a/libavfilter/vf_overlay_qsv.c
> > +++ b/libavfilter/vf_overlay_qsv.c
> > @@ -276,6 +276,7 @@ static int config_output(AVFilterLink *outlink)
> >  int ret;
> >
> >  av_log(ctx, AV_LOG_DEBUG, "Output is of %s.\n",
> > av_get_pix_fmt_name(outlink->format));
> > +vpp->qsv_param.out_sw_format = in0->format;
> >  if ((in0->format == AV_PIX_FMT_QSV && in1->format !=
> AV_PIX_FMT_QSV) ||
> >  (in0->format != AV_PIX_FMT_QSV && in1->format ==
> AV_PIX_FMT_QSV)) {
> >  av_log(ctx, AV_LOG_ERROR, "Mixing hardware and software pixel
> > formats is not supported.\n");
> > @@ -288,6 +289,7 @@ static int config_output(AVFilterLink *outlink)
> >  av_log(ctx, AV_LOG_ERROR, "Inputs with different underlying QSV
> > devices are forbidden.\n");
> >  return AVERROR(EINVAL);
> >  }
> > +vpp->qsv_param.out_sw_format = hw_frame0->sw_format;
> >  }
> >
> >  outlink->w  = vpp->var_values[VAR_MW];
> > --
> 
> Isn't this a bit too optimistic?
> 
> The doc says:
> 
> The only supported combinations of input and output color formats are:
> 
> RGB to RGB,
> NV12 to NV12,
> RGB and NV12 to NV12, for per pixel alpha blending use case.
> 
> https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-
> man.md#mfxextvppcomposite
> 
> Kind regards,
> softworkz

Thanks for your feedback.
We are checking this.

> 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V6 5/5] libavutil/hwcontext_vulkan: specify the modifier to create VKImage

[Chen, Wenbin]

> 7 Dec 2021, 10:05 by wenbin.c...@intel.com:
> 
> > When vulkan image exports to drm, the tilling need to be
> > VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to
> create vulkan
> > image using this format.
> >
> > Now the following command line works:
> >
> > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -
> hwaccel_output_format \
> > vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan,
> \
> > scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v
> h264_vaapi output.264
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_vulkan.c | 133
> +--
> >  1 file changed, 127 insertions(+), 6 deletions(-)
> >
> 
> Pushed the patchset, along with the transpose filter.
> Tested on all 3 vendors, works fine. AMD requires
> dedicated allocation, for which I added an error condition.
> 
> The code was quite ugly in quite a lot of places,
> so I did touch mostly every line of it in some way.
> 
> Please fix your quirky, limited hardware next time.

Thanks for your review and feedback.
This limitation of hardware also bothered me for some time.

Best Regards
Wenbin
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V2 1/3] libavcodec/vaapi_decode: fix the problem that init_pool_size < nb_surface

[Chen, Wenbin]

> Quoting Wenbin Chen (2021-11-30 09:44:04)
> > For vaapi if the init_pool_size is not zero, the pool size is fixed.
> > This means max surfaces is init_pool_size, but when mapping vaapi
> > frame to qsv frame, the init_pool_size < nb_surface. The cause is that
> > vaapi_decode_make_config() config the init_pool_size and it is called
> > twice. The first time is to init frame_context and the second time is to
> > init codec. On the second time the init_pool_size is changed to original
> > value so the init_pool_size is lower than the reall size because
> > pool_size used to initialize frame_context need to plus thread_count and
> > 3 (guarantee 4 base work surfaces). Now add code to make sure
> > init_pool_size is only set once. Now the following commandline works:
> >
> > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> > -hwaccel_output_format vaapi -i input.264 \
> > -vf "hwmap=derive_device=qsv,format=qsv" \
> > -c:v h264_qsv output.264
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavcodec/vaapi_decode.c | 34 ++
> >  1 file changed, 18 insertions(+), 16 deletions(-)
> >
> > diff --git a/libavcodec/vaapi_decode.c b/libavcodec/vaapi_decode.c
> > index 665af370ed..aab8162989 100644
> > --- a/libavcodec/vaapi_decode.c
> > +++ b/libavcodec/vaapi_decode.c
> > @@ -572,22 +572,24 @@ static int
> vaapi_decode_make_config(AVCodecContext *avctx,
> >  if (err < 0)
> >  goto fail;
> >
> > -frames->initial_pool_size = 1;
> > -// Add per-codec number of surfaces used for storing reference
> frames.
> > -switch (avctx->codec_id) {
> > -case AV_CODEC_ID_H264:
> > -case AV_CODEC_ID_HEVC:
> > -case AV_CODEC_ID_AV1:
> > -frames->initial_pool_size += 16;
> > -break;
> > -case AV_CODEC_ID_VP9:
> > -frames->initial_pool_size += 8;
> > -break;
> > -case AV_CODEC_ID_VP8:
> > -frames->initial_pool_size += 3;
> > -break;
> > -default:
> > -frames->initial_pool_size += 2;
> > +if (!frames->initial_pool_size) {
> > +frames->initial_pool_size = 1;
> > +// Add per-codec number of surfaces used for storing reference
> frames.
> > +switch (avctx->codec_id) {
> > +case AV_CODEC_ID_H264:
> > +case AV_CODEC_ID_HEVC:
> > +case AV_CODEC_ID_AV1:
> > +frames->initial_pool_size += 16;
> > +break;
> > +case AV_CODEC_ID_VP9:
> > +frames->initial_pool_size += 8;
> > +break;
> > +case AV_CODEC_ID_VP8:
> > +frames->initial_pool_size += 3;
> > +break;
> > +default:
> > +frames->initial_pool_size += 2;
> > +}
> 
> Seems to me that ff_vaapi_decode_init() should not pass hw_frames_ctx to
> vaapi_decode_make_config() at all, because it is already initialized and
> must not be modified.
> 
> --
> Anton Khirnov

You are right. I don't need to add an new check. ff_vaapi_decode_init() should 
pass NULL
to vaapi_decode_make_config. Thanks for your review. I will submit a new patch

Regards
Wenbin
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V5 5/5] libavutil/hwcontext_vulkan: specify the modifier to create VKImage

[Chen, Wenbin]

> 2 Dec 2021, 02:43 by wenbin.c...@intel.com:
> 
> > When vulkan image exports to drm, the tilling need to be
> > VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to
> create vulkan
> > image using this format.
> >
> > Now the following command line works:
> >
> > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -
> hwaccel_output_format \
> > vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan,
> \
> > scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v
> h264_vaapi output.264
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_vulkan.c | 127
> +--
> >  1 file changed, 121 insertions(+), 6 deletions(-)
> >
> > diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> > index f980b72720..8224c0d4e4 100644
> > --- a/libavutil/hwcontext_vulkan.c
> > +++ b/libavutil/hwcontext_vulkan.c
> > @@ -120,6 +120,9 @@ typedef struct VulkanFramesPriv {
> >  /* Image transfers */
> >  VulkanExecCtx upload_ctx;
> >  VulkanExecCtx download_ctx;
> > +
> > +/*modifier info*/
> > +VkImageDrmFormatModifierListCreateInfoEXT *modifier_info;
> >  } VulkanFramesPriv;
> >
> >  typedef struct AVVkFrameInternal {
> > @@ -242,6 +245,28 @@ const VkFormat *av_vkfmt_from_pixfmt(enum
> AVPixelFormat p)
> >  return NULL;
> >  }
> >
> > +static void *find_in_structure_list(VkBaseOutStructure *stru_list,
> VkStructureType sType) {
> > +if (!stru_list)
> > +return NULL;
> > +
> > +for(;stru_list;stru_list = stru_list->pNext)
> > +if (stru_list->sType == sType)
> > +return stru_list;
> > +
> > +return NULL;
> > +}
> > +
> > +static void append_to_structure_list(VkBaseOutStructure **stru_list,
> VkBaseOutStructure *added_stru) {
> > +VkBaseOutStructure *p;
> > +if (!*stru_list) {
> > +*stru_list = added_stru;
> > +return;
> > +}
> > +for(p = *stru_list; p->pNext; p = p->pNext);
> > +p->pNext = added_stru;
> > +return;
> > +}
> >
> 
> Could you tidy these 2 functions up? Also, code style issues, as usual.
> Take a look at libplacebo's functions or copy them, they're much
> neater: https://0x1.st/tx.txt
> 
> 
> >  static int pixfmt_is_supported(AVHWDeviceContext *dev_ctx, enum
> AVPixelFormat p,
> >  int linear)
> >  {
> > @@ -2094,6 +2119,10 @@ static void
> try_export_flags(AVHWFramesContext *hwfc,
> >  AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx;
> >  VulkanDevicePriv *p = hwfc->device_ctx->internal->priv;
> >  FFVulkanFunctions *vk = &p->vkfn;
> > +const int has_modifiers = hwctx->tiling ==
> VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT;
> > +int loop_count;
> > +VkImageDrmFormatModifierListCreateInfoEXT *modifier_info =
> find_in_structure_list(hwctx->create_pnext,
> > +
> VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INF
> O_EXT);
> >  VkExternalImageFormatProperties eprops = {
> >  .sType =
> VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR,
> >  };
> > @@ -2101,9 +2130,18 @@ static void
> try_export_flags(AVHWFramesContext *hwfc,
> >  .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2,
> >  .pNext = &eprops,
> >  };
> > +VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info
> = {
> > +.sType =
> VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER
> _INFO_EXT,
> > +.pNext = NULL,
> > +.pQueueFamilyIndices   = p->qfs,
> > +.queueFamilyIndexCount = p->num_qfs,
> > +.sharingMode   = p->num_qfs > 1 ?
> VK_SHARING_MODE_CONCURRENT :
> > +  
> > VK_SHARING_MODE_EXCLUSIVE,
> > +};
> >  VkPhysicalDeviceExternalImageFormatInfo enext = {
> >  .sType =
> VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO,
> >  .handleType = exp,
> > +.pNext = has_modifiers && modifier_info ? &phy_dev_mod_info :
> NULL,
> >  };
> >  VkPhysicalDeviceImageFormatInfo2 pinfo = {
> >  .sType  =
> VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2,
> > @@ -2115,11 +2153,16 @@ static void
> try_export_flags(AVHWFramesContext *hwfc,
> >  .flags  = VK_IMAGE_CREATE_ALIAS_BIT,
> >  };
> >
> > -ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx-
> >phys_dev,
> > -  &pinfo, &props);
> > -if (ret == VK_SUCCESS) {
> > -*iexp |= exp;
> > -*comp_handle_types |=
> eprops.externalMemoryProperties.compatibleHandleTypes;
> > +loop_count = has_modifiers && modifier_info ? modifier_info-
> >drmFormatModifierCount : 1;
> > +for (int i = 0; i < loop_count; i++) {
> > +if (has_modifiers && modifier_info)
> > +phy_dev_mod_info.drmFormatModifier = modifier_info-
> >pDrmFormatModifiers[i];
> > +ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx-
> >phys_dev,
> > +&pinfo, &props);
> > +if (ret == VK_SUCCESS) {
> > +*ie

Re: [FFmpeg-devel] [PATCH V5 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> 2 Dec 2021, 02:53 by wenbin.c...@intel.com:
> 
> >> The vaapi can import external frame, but the planes of the external
> >> frames should be in the same drm object. A new option
> "contiguous_planes"
> >> is added to device. This flag tells device to allocate places in one
> >> memory. When device is derived from vaapi this flag will be enabled.
> >> A new flag frame_flag is also added to AVVulkanFramesContext. User
> >> can use this flag to force enable or disable this behaviour.
> >> A new variable "offset "is added to AVVKFrame. It describe describe the
> >> offset from the memory currently bound to the VkImage.
> >>
> >> Signed-off-by: Wenbin Chen 
> >> ---
> >>  libavutil/hwcontext_vulkan.c | 68
> >> +++-
> >>  libavutil/hwcontext_vulkan.h | 24 +
> >>  2 files changed, 91 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> >> index a0437c9661..eef9009ae1 100644
> >> --- a/libavutil/hwcontext_vulkan.c
> >> +++ b/libavutil/hwcontext_vulkan.c
> >> @@ -103,8 +103,14 @@ typedef struct VulkanDevicePriv {
> >>  /* Settings */
> >>  int use_linear_images;
> >>
> >> +/* allocate planes in a contiguous memory */
> >> +int contiguous_planes;
> >> +
> >>  /* Nvidia */
> >>  int dev_is_nvidia;
> >> +
> >> +/* Intel */
> >> +int dev_is_intel;
> >>  } VulkanDevicePriv;
> >>
> >>  typedef struct VulkanFramesPriv {
> >> @@ -153,6 +159,8 @@ typedef struct AVVkFrameInternal {
> >>  av_free((void *)props);\
> >>  }
> >>
> >> +#define VKF_FLAG(x, f) (((x) & (~AV_VK_FRAME_FLAG_NONE)) & (f))
> >> +
> >>  static const struct {
> >>  enum AVPixelFormat pixfmt;
> >>  const VkFormat vkfmts[4];
> >> @@ -1374,6 +1382,13 @@ static int
> >> vulkan_device_create_internal(AVHWDeviceContext *ctx,
> >>  if (opt_d)
> >>  p->use_linear_images = strtol(opt_d->value, NULL, 10);
> >>
> >> +opt_d = av_dict_get(opts, "contiguous_planes", NULL, 0);
> >> +if (opt_d)
> >> +p->contiguous_planes = strtol(opt_d->value, NULL, 10);
> >> +else
> >> +p->contiguous_planes = -1;
> >> +
> >> +
> >>  hwctx->enabled_dev_extensions = dev_info.ppEnabledExtensionNames;
> >>  hwctx->nb_enabled_dev_extensions = dev_info.enabledExtensionCount;
> >>
> >> @@ -1425,6 +1440,8 @@ static int
> vulkan_device_init(AVHWDeviceContext
> >> *ctx)
> >>
> >>  p->dev_is_nvidia = (p->props.properties.vendorID == 0x10de);
> >>
> >> +p->dev_is_intel = (p->props.properties.vendorID == 0x8086);
> >> +
> >>  vk->GetPhysicalDeviceQueueFamilyProperties(hwctx->phys_dev,
> >> &queue_num, NULL);
> >>  if (!queue_num) {
> >>  av_log(ctx, AV_LOG_ERROR, "Failed to get queues!\n");
> >> @@ -1742,8 +1759,12 @@ static int
> alloc_bind_mem(AVHWFramesContext
> >> *hwfc, AVVkFrame *f,
> >>  AVHWDeviceContext *ctx = hwfc->device_ctx;
> >>  VulkanDevicePriv *p = ctx->internal->priv;
> >>  FFVulkanFunctions *vk = &p->vkfn;
> >> +AVVulkanFramesContext *hwfctx = hwfc->hwctx;
> >>  const int planes = av_pix_fmt_count_planes(hwfc->sw_format);
> >>  VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } };
> >> +VkMemoryRequirements memory_requirements = { 0 };
> >> +int mem_size = 0;
> >> +int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 };
> >>
> >>  AVVulkanDeviceContext *hwctx = ctx->hwctx;
> >>
> >> @@ -1771,6 +1792,19 @@ static int
> alloc_bind_mem(AVHWFramesContext
> >> *hwfc, AVVkFrame *f,
> >>  req.memoryRequirements.size =
> >> FFALIGN(req.memoryRequirements.size,
> >>  p-
> >> >props.properties.limits.minMemoryMapAlignment);
> >>
> >> +if (VKF_FLAG(hwfctx->flags,
> >> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) {
> >> +if (memory_requirements.size == 0) {
> >> +memory_requirements = req.memoryRequirements;
> >> +} else if (memory_requirements.memoryTypeBits !=
> >> req.memoryRequirements.memoryTypeBits) {
> >> +av_log(hwfc, AV_LOG_ERROR, "the param for each planes are
> not
> >> the same\n");
> >> +return AVERROR(EINVAL);
> >> +}
> >> +
> >> +mem_size_list[i] = req.memoryRequirements.size;
> >> +mem_size += mem_size_list[i];
> >> +continue;
> >> +}
> >> +
> >>  /* In case the implementation prefers/requires dedicated allocation */
> >>  use_ded_mem = ded_req.prefersDedicatedAllocation |
> >>  ded_req.requiresDedicatedAllocation;
> >> @@ -1792,6 +1826,29 @@ static int
> alloc_bind_mem(AVHWFramesContext
> >> *hwfc, AVVkFrame *f,
> >>  bind_info[i].memory = f->mem[i];
> >>  }
> >>
> >> +if (VKF_FLAG(hwfctx->flags,
> >> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) {
> >> +memory_requirements.size = mem_size;
> >> +
> >> +/* Allocate memory */
> >> +if ((err = alloc_mem(ctx, &memory_requirements,
> >> +f->tiling == VK_IMAGE_TILING_LINEAR ?
> >> +VK_MEMORY_PROPERTY_HOS

Re: [FFmpeg-devel] [PATCH V5 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> The vaapi can import external frame, but the planes of the external
> frames should be in the same drm object. A new option "contiguous_planes"
> is added to device. This flag tells device to allocate places in one
> memory. When device is derived from vaapi this flag will be enabled.
> A new flag frame_flag is also added to AVVulkanFramesContext. User
> can use this flag to force enable or disable this behaviour.
> A new variable "offset "is added to AVVKFrame. It describe describe the
> offset from the memory currently bound to the VkImage.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_vulkan.c | 68
> +++-
>  libavutil/hwcontext_vulkan.h | 24 +
>  2 files changed, 91 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> index a0437c9661..eef9009ae1 100644
> --- a/libavutil/hwcontext_vulkan.c
> +++ b/libavutil/hwcontext_vulkan.c
> @@ -103,8 +103,14 @@ typedef struct VulkanDevicePriv {
>  /* Settings */
>  int use_linear_images;
> 
> +/* allocate planes in a contiguous memory */
> +int contiguous_planes;
> +
>  /* Nvidia */
>  int dev_is_nvidia;
> +
> +/* Intel */
> +int dev_is_intel;
>  } VulkanDevicePriv;
> 
>  typedef struct VulkanFramesPriv {
> @@ -153,6 +159,8 @@ typedef struct AVVkFrameInternal {
>  av_free((void *)props);  
>   \
>  }
> 
> +#define VKF_FLAG(x, f) (((x) & (~AV_VK_FRAME_FLAG_NONE)) & (f))
> +
>  static const struct {
>  enum AVPixelFormat pixfmt;
>  const VkFormat vkfmts[4];
> @@ -1374,6 +1382,13 @@ static int
> vulkan_device_create_internal(AVHWDeviceContext *ctx,
>  if (opt_d)
>  p->use_linear_images = strtol(opt_d->value, NULL, 10);
> 
> +opt_d = av_dict_get(opts, "contiguous_planes", NULL, 0);
> +if (opt_d)
> +p->contiguous_planes = strtol(opt_d->value, NULL, 10);
> +else
> +p->contiguous_planes = -1;
> +
> +
>  hwctx->enabled_dev_extensions = dev_info.ppEnabledExtensionNames;
>  hwctx->nb_enabled_dev_extensions = dev_info.enabledExtensionCount;
> 
> @@ -1425,6 +1440,8 @@ static int vulkan_device_init(AVHWDeviceContext
> *ctx)
> 
>  p->dev_is_nvidia = (p->props.properties.vendorID == 0x10de);
> 
> +p->dev_is_intel = (p->props.properties.vendorID == 0x8086);
> +
>  vk->GetPhysicalDeviceQueueFamilyProperties(hwctx->phys_dev,
> &queue_num, NULL);
>  if (!queue_num) {
>  av_log(ctx, AV_LOG_ERROR, "Failed to get queues!\n");
> @@ -1742,8 +1759,12 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
>  AVHWDeviceContext *ctx = hwfc->device_ctx;
>  VulkanDevicePriv *p = ctx->internal->priv;
>  FFVulkanFunctions *vk = &p->vkfn;
> +AVVulkanFramesContext *hwfctx = hwfc->hwctx;
>  const int planes = av_pix_fmt_count_planes(hwfc->sw_format);
>  VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } };
> +VkMemoryRequirements memory_requirements = { 0 };
> +int mem_size = 0;
> +int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 };
> 
>  AVVulkanDeviceContext *hwctx = ctx->hwctx;
> 
> @@ -1771,6 +1792,19 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
>  req.memoryRequirements.size =
> FFALIGN(req.memoryRequirements.size,
>p-
> >props.properties.limits.minMemoryMapAlignment);
> 
> +if (VKF_FLAG(hwfctx->flags,
> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) {
> +if (memory_requirements.size == 0) {
> +memory_requirements = req.memoryRequirements;
> +} else if (memory_requirements.memoryTypeBits !=
> req.memoryRequirements.memoryTypeBits) {
> +av_log(hwfc, AV_LOG_ERROR, "the param for each planes are not
> the same\n");
> +return AVERROR(EINVAL);
> +}
> +
> +mem_size_list[i] = req.memoryRequirements.size;
> +mem_size += mem_size_list[i];
> +continue;
> +}
> +
>  /* In case the implementation prefers/requires dedicated allocation 
> */
>  use_ded_mem = ded_req.prefersDedicatedAllocation |
>ded_req.requiresDedicatedAllocation;
> @@ -1792,6 +1826,29 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
>  bind_info[i].memory = f->mem[i];
>  }
> 
> +if (VKF_FLAG(hwfctx->flags,
> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) {
> +memory_requirements.size = mem_size;
> +
> +/* Allocate memory */
> +if ((err = alloc_mem(ctx, &memory_requirements,
> +f->tiling == VK_IMAGE_TILING_LINEAR ?
> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT :
> +VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
> +(void *)(((uint8_t *)alloc_pnext)),
> +&f->f

Re: [FFmpeg-devel] [PATCH V4 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> Quoting Wenbin Chen (2021-11-30 07:28:13)
> > diff --git a/libavutil/hwcontext_vulkan.h b/libavutil/hwcontext_vulkan.h
> > index fdf2a60156..c485ee7437 100644
> > --- a/libavutil/hwcontext_vulkan.h
> > +++ b/libavutil/hwcontext_vulkan.h
> > @@ -35,6 +35,17 @@
> >   * with the data pointer set to an AVVkFrame.
> >   */
> >
> > +/**
> > + * Defines the behaviour of frame allocation
> > + * AV_VK_FRAME_FLAG_NONE: planes will be allocated in separte
> memory
> > + * AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY: planes will be allocated
> in a
> > + * contiguous memory.
> > + */
> > +typedef enum {
> > +AV_VK_FRAME_FLAG_NONE = (1ULL << 0),
> > +AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY = (1ULL << 1) | 1ULL
> > +} AVVkFrameFlags;
> > +
> >  /**
> >   * Main Vulkan context, allocated as AVHWDeviceContext.hwctx.
> >   * All of these can be set before init to change what the context uses
> > @@ -157,6 +168,14 @@ typedef struct AVVulkanFramesContext {
> >   */
> >  void *create_pnext;
> >
> > +/**
> > + * Is a combination of AVVkFrameFlags. Defines the behaviour of frame
> > + * allocation.
> > + * If no flag is set, then the flags are automatically determined
> > + * based on the device.
> > + */
> > +int flags;
> > +
> >  /**
> >   * Extension data for memory allocation. Must have as many entries as
> >   * the number of planes of the sw_format.
> > @@ -198,6 +217,11 @@ typedef struct AVVkFrame {
> >  VkDeviceMemory mem[AV_NUM_DATA_POINTERS];
> >  size_t size[AV_NUM_DATA_POINTERS];
> >
> > +/**
> > + * Describe the offset from the memory currently bound to the VkImage.
> > + */
> > +size_t offset[AV_NUM_DATA_POINTERS];
> > +
> 
> These are public structs, you have to add any new fields at the end or
> you will break ABI compatibility.
> 
> --
> Anton Khirnov

Will resubmit the patchset. Thanks

Wenbin

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V4 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> The vaapi can import external frame, but the planes of the external
> frames should be in the same drm object. A new option "contiguous_planes"
> is added to device. This flag tells device to allocate places in one
> memory. When device is derived from vaapi this flag will be enabled.
> A new flag frame_flag is also added to AVVulkanFramesContext. User
> can use this flag to force enable or disable this behaviour.
> A new variable "offset "is added to AVVKFrame. It describe describe the
> offset from the memory currently bound to the VkImage.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_vulkan.c | 68
> +++-
>  libavutil/hwcontext_vulkan.h | 24 +
>  2 files changed, 91 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> index a0437c9661..eef9009ae1 100644
> --- a/libavutil/hwcontext_vulkan.c
> +++ b/libavutil/hwcontext_vulkan.c
> @@ -103,8 +103,14 @@ typedef struct VulkanDevicePriv {
>  /* Settings */
>  int use_linear_images;
> 
> +/* allocate planes in a contiguous memory */
> +int contiguous_planes;
> +
>  /* Nvidia */
>  int dev_is_nvidia;
> +
> +/* Intel */
> +int dev_is_intel;
>  } VulkanDevicePriv;
> 
>  typedef struct VulkanFramesPriv {
> @@ -153,6 +159,8 @@ typedef struct AVVkFrameInternal {
>  av_free((void *)props);  
>   \
>  }
> 
> +#define VKF_FLAG(x, f) (((x) & (~AV_VK_FRAME_FLAG_NONE)) & (f))
> +
>  static const struct {
>  enum AVPixelFormat pixfmt;
>  const VkFormat vkfmts[4];
> @@ -1374,6 +1382,13 @@ static int
> vulkan_device_create_internal(AVHWDeviceContext *ctx,
>  if (opt_d)
>  p->use_linear_images = strtol(opt_d->value, NULL, 10);
> 
> +opt_d = av_dict_get(opts, "contiguous_planes", NULL, 0);
> +if (opt_d)
> +p->contiguous_planes = strtol(opt_d->value, NULL, 10);
> +else
> +p->contiguous_planes = -1;
> +
> +
>  hwctx->enabled_dev_extensions = dev_info.ppEnabledExtensionNames;
>  hwctx->nb_enabled_dev_extensions = dev_info.enabledExtensionCount;
> 
> @@ -1425,6 +1440,8 @@ static int vulkan_device_init(AVHWDeviceContext
> *ctx)
> 
>  p->dev_is_nvidia = (p->props.properties.vendorID == 0x10de);
> 
> +p->dev_is_intel = (p->props.properties.vendorID == 0x8086);
> +
>  vk->GetPhysicalDeviceQueueFamilyProperties(hwctx->phys_dev,
> &queue_num, NULL);
>  if (!queue_num) {
>  av_log(ctx, AV_LOG_ERROR, "Failed to get queues!\n");
> @@ -1742,8 +1759,12 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
>  AVHWDeviceContext *ctx = hwfc->device_ctx;
>  VulkanDevicePriv *p = ctx->internal->priv;
>  FFVulkanFunctions *vk = &p->vkfn;
> +AVVulkanFramesContext *hwfctx = hwfc->hwctx;
>  const int planes = av_pix_fmt_count_planes(hwfc->sw_format);
>  VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } };
> +VkMemoryRequirements memory_requirements = { 0 };
> +int mem_size = 0;
> +int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 };
> 
>  AVVulkanDeviceContext *hwctx = ctx->hwctx;
> 
> @@ -1771,6 +1792,19 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
>  req.memoryRequirements.size =
> FFALIGN(req.memoryRequirements.size,
>p-
> >props.properties.limits.minMemoryMapAlignment);
> 
> +if (VKF_FLAG(hwfctx->flags,
> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) {
> +if (memory_requirements.size == 0) {
> +memory_requirements = req.memoryRequirements;
> +} else if (memory_requirements.memoryTypeBits !=
> req.memoryRequirements.memoryTypeBits) {
> +av_log(hwfc, AV_LOG_ERROR, "the param for each planes are not
> the same\n");
> +return AVERROR(EINVAL);
> +}
> +
> +mem_size_list[i] = req.memoryRequirements.size;
> +mem_size += mem_size_list[i];
> +continue;
> +}
> +
>  /* In case the implementation prefers/requires dedicated allocation 
> */
>  use_ded_mem = ded_req.prefersDedicatedAllocation |
>ded_req.requiresDedicatedAllocation;
> @@ -1792,6 +1826,29 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
>  bind_info[i].memory = f->mem[i];
>  }
> 
> +if (VKF_FLAG(hwfctx->flags,
> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY)) {
> +memory_requirements.size = mem_size;
> +
> +/* Allocate memory */
> +if ((err = alloc_mem(ctx, &memory_requirements,
> +f->tiling == VK_IMAGE_TILING_LINEAR ?
> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT :
> +VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
> +(void *)(((uint8_t *)alloc_pnext)),
> +&f->f

Re: [FFmpeg-devel] [PATCH 3/3] libavutil/hwcontext_opencl: fix a bug for mapping qsv frame to opencl

[Chen, Wenbin]

> Quoting Wenbin Chen (2021-11-16 09:16:23)
> > From: nyanmisaka 
> >
> > mfxHDLPair was added to qsv, so modify qsv->opencl map function as well.
> > Now the following commandline works:
> >
> > ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 \
> > -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -
> filter_hw_device ocl \
> > -hwaccel qsv -hwaccel_output_format qsv -hwaccel_device qs -c:v
> h264_qsv \
> > -i input.264 -vf
> "hwmap=derive_device=opencl,format=opencl,avgblur_opencl, \
> > hwmap=derive_device=qsv:reverse=1:extra_hw_frames=32,format=qsv" \
> > -c:v h264_qsv output.264
> >
> > Signed-off-by: nyanmisaka 
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_opencl.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavutil/hwcontext_opencl.c b/libavutil/hwcontext_opencl.c
> > index 26a3a24593..4b6e74ff6f 100644
> > --- a/libavutil/hwcontext_opencl.c
> > +++ b/libavutil/hwcontext_opencl.c
> > @@ -2249,7 +2249,8 @@ static int
> opencl_map_from_qsv(AVHWFramesContext *dst_fc, AVFrame *dst,
> >  #if CONFIG_LIBMFX
> >  if (src->format == AV_PIX_FMT_QSV) {
> >  mfxFrameSurface1 *mfx_surface = (mfxFrameSurface1*)src->data[3];
> > -va_surface = *(VASurfaceID*)mfx_surface->Data.MemId;
> > +mfxHDLPair *pair = (mfxHDLPair*)mfx_surface->Data.MemId;
> > +va_surface = *(VASurfaceID*)pair->first;
> >  } else
> >  #endif
> 
> The casts here are starting to look like overly arcane black magic.
> Who is responsible for setting MemId here? I assume it's something in
> hwcontext_qsv, but is that guaranteed?
> 
> It would be better for hwcontext_qsv to abstract reading/writing MemId
> contents into a function/macro, so other code can just call it and not
> hardcode internal implementation details.
> 
> --
> Anton Khirnov

Thanks for review. I will resubmit the patchset

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V3 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> 26 Nov 2021, 03:54 by wenbin.c...@intel.com:
> 
> > The vaapi can import external frame, but the planes of the external
> > frames should be in the same drm object. A new option
> "contiguous_planes"
> > is added to device. This flag tells device to allocate places in one
> > memory. When device is derived from vaapi this flag will be enabled.
> > A new flag frame_flag is also added to AVVulkanFramesContext. User
> > can use this flag to force enable or disable this behaviour.
> > A new variable "offset "is added to AVVKFrame. It describe describe the
> > offset from the memory currently bound to the VkImage.
> >
> > Signed-off-by: Wenbin Chen 
> >
> 
> Why is a new offset variable needed?
> vkGetImageSubresourceLayout is valid for DRM tiled images.
> According to the specs,
> "If the image’s tiling is VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT,
> then vkGetImageSubresourceLayout describes one memory plane of the
> image. If the image’s tiling is
> VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT and the image is non-
> linear, then the returned layout has an implementation-dependent meaning;
> the vendor of the image’s DRM format modifier may provide documentation
> that explains how to interpret the returned layout.".
> 
> Isn't this what you already have in the offset field?

The offset we get from vkGetImageSubresourceLayout is from the start of the 
image or plane.
The offset drm_object need is from the start of the memory, and 
vkGetImageSubresourceLayout
cannot get this information. I add a new offset variable because I allocate all 
planes in one memory 
not because I use VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT.

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V2 3/5] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> 24 Nov 2021, 06:28 by wenbin.c...@intel.com:
> 
> > The vaapi can import external frame, but the planes of the external
> > frames should be in the same drm object. A new option
> "contiguous_planes"
> > is added to device. This flag tells device to allocate places in one
> > memory. When device is derived from vaapi this flag will be enabled.
> > A new flag frame_flag is also added to AVVulkanFramesContext. User
> > can use this flag to force enable or disable this behaviour.
> > A new variable "offset "is added to AVVKFrame. It describe describe the
> > offset from the memory currently bound to the VkImage.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_vulkan.c | 62
> ++--
> >  libavutil/hwcontext_vulkan.h | 22 +
> >  2 files changed, 82 insertions(+), 2 deletions(-)
> >
> > diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> > index f1e750cd3e..4100e8b0a2 100644
> > --- a/libavutil/hwcontext_vulkan.c
> > +++ b/libavutil/hwcontext_vulkan.c
> > @@ -103,6 +103,9 @@ typedef struct VulkanDevicePriv {
> >  /* Settings */
> >  int use_linear_images;
> >
> > +/* allocate planes in a contiguous memory */
> > +int contiguous_planes;
> > +
> >  /* Nvidia */
> >  int dev_is_nvidia;
> >
> 
> Add a new `int dev_is_intel;` field, and set it
> in `vulkan_device_init()`, where `dev_is_nvidia` is set.
> 
> 
> >  } VulkanDevicePriv;
> > @@ -1266,6 +1269,11 @@ static int
> vulkan_device_create_internal(AVHWDeviceContext *ctx,
> >  if (opt_d)
> >  p->use_linear_images = strtol(opt_d->value, NULL, 10);
> >
> > +opt_d = av_dict_get(opts, "contiguous_planes", NULL, 0);
> > +if (opt_d)
> > +p->contiguous_planes = strtol(opt_d->value, NULL, 10);
> >
> 
> Set `p->contiguous_planes` to -1 if not specified.
> 
> 
> >  hwctx->enabled_dev_extensions = dev_info.ppEnabledExtensionNames;
> >  hwctx->nb_enabled_dev_extensions = dev_info.enabledExtensionCount;
> >
> > @@ -1410,8 +1418,10 @@ static int
> vulkan_device_derive(AVHWDeviceContext *ctx,
> >  return AVERROR_EXTERNAL;
> >  }
> >
> > -if (strstr(vendor, "Intel"))
> > +if (strstr(vendor, "Intel")) {
> > +av_dict_set_int(&opts, "contiguous_planes", 1, 0);
> >
> 
> Don't set this here, it's not needed with the changes I mentioned.
> 
> 
> >  dev_select.vendor_id = 0x8086;
> > +}
> >  if (strstr(vendor, "AMD"))
> >  dev_select.vendor_id = 0x1002;
> >
> > @@ -1634,8 +1644,12 @@ static int
> alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
> >  AVHWDeviceContext *ctx = hwfc->device_ctx;
> >  VulkanDevicePriv *p = ctx->internal->priv;
> >  FFVulkanFunctions *vk = &p->vkfn;
> > +AVVulkanFramesContext *hwfctx = hwfc->hwctx;
> >  const int planes = av_pix_fmt_count_planes(hwfc->sw_format);
> >  VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } };
> > +VkMemoryRequirements memory_requirements = { 0 };
> > +int mem_size = 0;
> > +int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 };
> >
> >  AVVulkanDeviceContext *hwctx = ctx->hwctx;
> >
> > @@ -1663,6 +1677,19 @@ static int
> alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
> >  req.memoryRequirements.size = FFALIGN(req.memoryRequirements.size,
> >  p->props.properties.limits.minMemoryMapAlignment);
> >
> > +if (hwfctx->contiguous_planes ==
> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY) {
> >
> 
> Bitwise &. Not equals.
> 
> 
> > +if (memory_requirements.size == 0) {
> > +memory_requirements = req.memoryRequirements;
> > +} else if (memory_requirements.memoryTypeBits !=
> req.memoryRequirements.memoryTypeBits) {
> > +av_log(hwfc, AV_LOG_ERROR, "the param for each planes are
> not the same\n");
> > +return AVERROR(EINVAL);
> > +}
> > +
> > +mem_size_list[i] = req.memoryRequirements.size;
> > +mem_size += mem_size_list[i];
> > +continue;
> > +}
> > +
> >  /* In case the implementation prefers/requires dedicated allocation */
> >  use_ded_mem = ded_req.prefersDedicatedAllocation |
> >  ded_req.requiresDedicatedAllocation;
> > @@ -1684,6 +1711,29 @@ static int
> alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
> >  bind_info[i].memory = f->mem[i];
> >  }
> >
> > +if (hwfctx->contiguous_planes ==
> AV_VK_FRAME_FLAG_CONTIGUOUS_MEMORY) {
> >
> 
> Same.
> 
> 
> > +memory_requirements.size = mem_size;
> > +
> > +/* Allocate memory */
> > +if ((err = alloc_mem(ctx, &memory_requirements,
> > +f->tiling == VK_IMAGE_TILING_LINEAR ?
> > +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT :
> > +VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
> > +(void *)(((uint8_t *)alloc_pnext)),
> > +&f->flags, &f->mem[0])))
> > +return err;
> > +
> > +f->size[0] = memory_requi

Re: [FFmpeg-devel] [PATCH V2 1/5] hwcontext_vaapi: Use PRIME_2 memory type for modifiers.

[Chen, Wenbin]

> 24 Nov 2021, 06:28 by wenbin.c...@intel.com:
> 
> > From: Bas Nieuwenhuizen 
> >
> > This way we can pass explicit modifiers in. Sometimes the
> > modifier matters for the number of memory planes that
> > libva accepts, in particular when dealing with
> > driver-compressed textures. Furthermore the driver might
> > not actually be able to determine the implicit modifier
> > if all the buffer-passing has used explicit modifier.
> > All these issues should be resolved by passing in the
> > modifier, and for that we switch to using the PRIME_2
> > memory type.
> >
> > Tested with experimental radeonsi patches for modifiers
> > and kmsgrab. Also tested with radeonsi without the
> > patches to double-check it works without PRIME_2 support.
> >
> > v2:
> >  Cache PRIME_2 support to avoid doing two calls every time on
> >  libva drivers that do not support it.
> >
> > v3:
> >  Remove prime2_vas usage.
> >
> 
> I've pinged jkqxz, the maintainer, but to me it looks good,
> and considering how long this has been on the ML, I'll
> apply it alongside the Vulkan patches if he doesn't respond.
> By the way, it mentions the author is Bas Nieuwenhuizen,
> should I change the commit author when I apply?

Please change commit author to Bas Nieuwenhuizen.
Thanks.

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 3/3] libavutil/hwcontext_opencl: fix a bug for mapping qsv frame to opencl

[Chen, Wenbin]

> From: nyanmisaka 
> 
> mfxHDLPair was added to qsv, so modify qsv->opencl map function as well.
> Now the following commandline works:
> 
> ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 \
> -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -
> filter_hw_device ocl \
> -hwaccel qsv -hwaccel_output_format qsv -hwaccel_device qs -c:v h264_qsv
> \
> -i input.264 -vf
> "hwmap=derive_device=opencl,format=opencl,avgblur_opencl, \
> hwmap=derive_device=qsv:reverse=1:extra_hw_frames=32,format=qsv" \
> -c:v h264_qsv output.264
> 
> Signed-off-by: nyanmisaka 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_opencl.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_opencl.c b/libavutil/hwcontext_opencl.c
> index 26a3a24593..4b6e74ff6f 100644
> --- a/libavutil/hwcontext_opencl.c
> +++ b/libavutil/hwcontext_opencl.c
> @@ -2249,7 +2249,8 @@ static int
> opencl_map_from_qsv(AVHWFramesContext *dst_fc, AVFrame *dst,
>  #if CONFIG_LIBMFX
>  if (src->format == AV_PIX_FMT_QSV) {
>  mfxFrameSurface1 *mfx_surface = (mfxFrameSurface1*)src->data[3];
> -va_surface = *(VASurfaceID*)mfx_surface->Data.MemId;
> +mfxHDLPair *pair = (mfxHDLPair*)mfx_surface->Data.MemId;
> +va_surface = *(VASurfaceID*)pair->first;
>  } else
>  #endif
>  if (src->format == AV_PIX_FMT_VAAPI) {
> --
> 2.25.1
> 

ping

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 2/3] libavutil/hwcontext_qsv: fix a bug for mapping vaapi frame to qsv

[Chen, Wenbin]

> From: nyanmisaka 
> 
> The data stored in data[3] in VAAPI AVFrame is VASurfaceID while
> the data stored in pair->first is the pointer of VASurfaceID, so
> we need to do cast to make following commandline works:
> 
> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> -hwaccel_output_format vaapi -i input.264 \
> -vf "hwmap=derive_device=qsv,format=qsv" -c:v h264_qsv output.264
> 
> Signed-off-by: nyanmisaka 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_qsv.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> index c18747f7eb..d83754193a 100644
> --- a/libavutil/hwcontext_qsv.c
> +++ b/libavutil/hwcontext_qsv.c
> @@ -1220,7 +1220,7 @@ static int qsv_map_to(AVHWFramesContext
> *dst_ctx,
>  case AV_PIX_FMT_VAAPI:
>  {
>  mfxHDLPair *pair = (mfxHDLPair*)hwctx->surfaces[i].Data.MemId;
> -if (pair->first == src->data[3]) {
> +if (*(VASurfaceID*)pair->first == (VASurfaceID)src->data[3]) {
>  index = i;
>  break;
>  }
> --
> 2.25.1
> 

ping

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] libavcodec/vaapi_decode: fix the problem that init_pool_size < nb_surface

[Chen, Wenbin]

> For vaapi if the init_pool_size is not zero, the pool size is fixed.
> This means max surfaces is init_pool_size, but when mapping vaapi
> frame to qsv frame, the init_pool_size < nb_surface. The cause is that
> vaapi_decode_make_config() config the init_pool_size and it is called
> twice. The first time is to init frame_context and the second time is to
> init codec. On the second time the init_pool_size is changed to original
> value so the init_pool_size is lower than the reall size because
> pool_size used to initialize frame_context need to plus thread_count and
> 3 (guarantee 4 base work surfaces). Now add code to make sure
> init_pool_size is only set once. Now the following commandline works:
> 
> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> -hwaccel_output_format vaapi -i input.264 \
> -vf "hwmap=derive_device=qsv,format=qsv" \
> -c:v h264_qsv output.264
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/vaapi_decode.c | 34 ++
>  1 file changed, 18 insertions(+), 16 deletions(-)
> 
> diff --git a/libavcodec/vaapi_decode.c b/libavcodec/vaapi_decode.c
> index 665af370ed..aab8162989 100644
> --- a/libavcodec/vaapi_decode.c
> +++ b/libavcodec/vaapi_decode.c
> @@ -572,22 +572,24 @@ static int
> vaapi_decode_make_config(AVCodecContext *avctx,
>  if (err < 0)
>  goto fail;
> 
> -frames->initial_pool_size = 1;
> -// Add per-codec number of surfaces used for storing reference 
> frames.
> -switch (avctx->codec_id) {
> -case AV_CODEC_ID_H264:
> -case AV_CODEC_ID_HEVC:
> -case AV_CODEC_ID_AV1:
> -frames->initial_pool_size += 16;
> -break;
> -case AV_CODEC_ID_VP9:
> -frames->initial_pool_size += 8;
> -break;
> -case AV_CODEC_ID_VP8:
> -frames->initial_pool_size += 3;
> -break;
> -default:
> -frames->initial_pool_size += 2;
> +if (!frames->initial_pool_size) {
> +frames->initial_pool_size = 1;
> +// Add per-codec number of surfaces used for storing reference
> frames.
> +switch (avctx->codec_id) {
> +case AV_CODEC_ID_H264:
> +case AV_CODEC_ID_HEVC:
> +case AV_CODEC_ID_AV1:
> +frames->initial_pool_size += 16;
> +break;
> +case AV_CODEC_ID_VP9:
> +frames->initial_pool_size += 8;
> +break;
> +case AV_CODEC_ID_VP8:
> +frames->initial_pool_size += 3;
> +break;
> +default:
> +frames->initial_pool_size += 2;
> +}
>  }
>  }
> 
> --
> 2.25.1
> 

ping

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3 1/1] avutils/hwcontext: When deriving a hwdevice, search for existing device in both directions

[Chen, Wenbin]

> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> > Xiang, Haihao
> > Sent: Monday, October 18, 2021 6:48 AM
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH v3 1/1] avutils/hwcontext: When
> > deriving a hwdevice, search for existing device in both directions
> >
> > On Mon, 2021-10-11 at 04:19 +, Soft Works wrote:
> > > The test /libavutil/tests/hwdevice checks that when deriving a
> > device
> > > from a source device and then deriving back to the type of the
> > source
> > > device, the result is matching the original source device, i.e. the
> > > derivation mechanism doesn't create a new device in this case.
> > >
> > > Previously, this test was usually passed, but only due to two
> > different
> > > kind of flaws:
> > >
> > > 1. The test covers only a single level of derivation (and back)
> > >
> > > It derives device Y from device X and then Y back to the type of X
> > and
> > > checks whether the result matches X.
> > >
> > > What it doesn't check for, are longer chains of derivation like:
> > >
> > > CUDA1 > OpenCL2 > CUDA3 and then back to OpenCL4
> > >
> > > In that case, the second derivation returns the first device (CUDA3
> > ==
> > > CUDA1), but when deriving OpenCL4, hwcontext.c was creating a new
> > > OpenCL4 context instead of returning OpenCL2, because there was no
> > link
> > > from CUDA1 to OpenCL2 (only backwards from OpenCL2 to CUDA1)
> > >
> > > If the test would check for two levels of derivation, it would have
> > > failed.
> > >
> > > This patch fixes those (yet untested) cases by introducing forward
> > > references (derived_device) in addition to the existing back
> > references
> > > (source_device).
> > >
> > > 2. hwcontext_qsv didn't properly set the source_device
> > >
> > > In case of QSV, hwcontext_qsv creates a source context internally
> > > (vaapi, dxva2 or d3d11va) without calling
> > av_hwdevice_ctx_create_derived
> > > and without setting source_device.
> > >
> > > This way, the hwcontext test ran successful, but what practically
> > > happened, was that - for example - deriving vaapi from qsv didn't
> > return
> > > the original underlying vaapi device and a new one was created
> > instead:
> > > Exactly what the test is intended to detect and prevent. It just
> > > couldn't do so, because the original device was hidden (= not set
> > as the
> > > source_device of the QSV device).
> > >
> > > This patch properly makes these setting and fixes all derivation
> > > scenarios.
> > >
> > > (at a later stage, /libavutil/tests/hwdevice should be extended to
> > check
> > > longer derivation chains as well)
> > >
> > > Signed-off-by: softworkz 
> > > ---
> > > v3: avoid double-release as suggested by Haihao
> > >
> > >  libavutil/hwcontext.c  | 38
> > ++
> > >  libavutil/hwcontext.h  |  1 +
> > >  libavutil/hwcontext_internal.h |  6 ++
> > >  libavutil/hwcontext_qsv.c  | 16 ++
> > >  4 files changed, 57 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/libavutil/hwcontext.c b/libavutil/hwcontext.c
> > > index 31c7840dba..1a50635018 100644
> > > --- a/libavutil/hwcontext.c
> > > +++ b/libavutil/hwcontext.c
> > > @@ -122,6 +122,7 @@ static const AVClass hwdevice_ctx_class = {
> > >  static void hwdevice_ctx_free(void *opaque, uint8_t *data)
> > >  {
> > >  AVHWDeviceContext *ctx = (AVHWDeviceContext*)data;
> > > +int i;
> > >
> > >  /* uninit might still want access the hw context and the user
> > >   * free() callback might destroy it, so uninit has to be
> > called first */
> > > @@ -132,6 +133,8 @@ static void hwdevice_ctx_free(void *opaque,
> > uint8_t *data)
> > >  ctx->free(ctx);
> > >
> > >  av_buffer_unref(&ctx->internal->source_device);
> > > +for (i = 0; i < AV_HWDEVICE_TYPE_NB; i++)
> > > +av_buffer_unref(&ctx->internal->derived_devices[i]);
> > >
> > >  av_freep(&ctx->hwctx);
> > >  av_freep(&ctx->internal->priv);
> > > @@ -643,6 +646,26 @@ fail:
> > >  return ret;
> > >  }
> > >
> > > +static AVBufferRef* find_derived_hwdevice_ctx(AVBufferRef
> > *src_ref, enum
> > > AVHWDeviceType type)
> > > +{
> > > +AVBufferRef *tmp_ref;
> > > +AVHWDeviceContext *src_ctx;
> > > +int i;
> > > +
> > > +src_ctx = (AVHWDeviceContext*)src_ref->data;
> > > +if (src_ctx->type == type)
> > > +return src_ref;
> > > +
> > > +for (i = 0; i < AV_HWDEVICE_TYPE_NB; i++)
> > > +if (src_ctx->internal->derived_devices[i]) {
> > > +tmp_ref = find_derived_hwdevice_ctx(src_ctx->internal-
> > > >derived_devices[i], type);
> > > +if (tmp_ref)
> > > +return tmp_ref;
> > > +}
> > > +
> > > +return NULL;
> > > +}
> > > +
> > >  int av_hwdevice_ctx_create_derived_opts(AVBufferRef **dst_ref_ptr,
> > >  enum AVHWDeviceType type,
> > >  AVBufferRef *src_ref,
> > > 

Re: [FFmpeg-devel] [PATCH 4/7] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> 19 Nov 2021, 19:13 by d...@lynne.ee:
> 
> > 19 Nov 2021, 18:59 by d...@lynne.ee:
> >
> >> 15 Nov 2021, 08:25 by wenbin.c...@intel.com:
> >>
>  9 Nov 2021, 10:18 by wenbin.c...@intel.com:
> 
>  > The vaapi can import external frame, but the planes of the external
>  > frames should be in the same drm object. I add a new function to
>  > allocate vkFrame in one memory and vulkan device will choose a way
>  > to allocate memory according to one_memory flag.
>  > A new variable is added to AVVKFrame to store the offset of each
> plane.
>  >
>  > Signed-off-by: Wenbin Chen 
>  > ---
>  >  libavutil/hwcontext_vulkan.c | 46
>  +++-
>  >  libavutil/hwcontext_vulkan.h |  1 +
>  >  2 files changed, 46 insertions(+), 1 deletion(-)
>  >
>  > diff --git a/libavutil/hwcontext_vulkan.c
> b/libavutil/hwcontext_vulkan.c
>  > index ccf3e58f49..f7878ed9c3 100644
>  > --- a/libavutil/hwcontext_vulkan.c
>  > +++ b/libavutil/hwcontext_vulkan.c
>  > @@ -1600,6 +1600,9 @@ static int
> alloc_bind_mem(AVHWFramesContext
>  *hwfc, AVVkFrame *f,
>  >  FFVulkanFunctions *vk = &p->vkfn;
>  >  const int planes = av_pix_fmt_count_planes(hwfc->sw_format);
>  >  VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] =
> { { 0 } };
>  > +VkMemoryRequirements memory_requirements = { 0 };
>  > +int mem_size = 0;
>  > +int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 };
>  >
>  >  AVVulkanDeviceContext *hwctx = ctx->hwctx;
>  >
>  > @@ -1627,6 +1630,23 @@ static int
>  alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
>  >  req.memoryRequirements.size =
> FFALIGN(req.memoryRequirements.size,
>  >  p->props.properties.limits.minMemoryMapAlignment);
>  >
>  > +if (p->use_one_memory) {
>  > +if (ded_req.prefersDedicatedAllocation |
>  ded_req.requiresDedicatedAllocation) {
>  > +av_log(hwfc, AV_LOG_ERROR, "Cannot use dedicated
> allocation
>  for intel vaapi\n");
>  > +return AVERROR(EINVAL);
>  > +}
>  >
> 
>  We don't set the flag unless the driver tells us to, so if the
>  driver asks us to use dedicated memory when it can't handle such
>  images, shouldn't the driver just not set this flag?
> 
> >>>
> >>> I check the dedicatedAllocation flag because I don't know if vaapi driver
> >>> support importing dedicated memory.
> >>> Actually I am not sure if I need to check this flag for vaapi. I can 
> >>> remove it.
> >>>
> 
> 
>  > +if (memory_requirements.size == 0) {
>  > +memory_requirements = req.memoryRequirements;
>  > +} else if (memory_requirements.memoryTypeBits !=
>  req.memoryRequirements.memoryTypeBits) {
>  > +av_log(hwfc, AV_LOG_ERROR, "the param for each planes
> are
>  not the same\n");
>  > +return AVERROR(EINVAL);
>  > +}
>  > +
>  > +mem_size_list[i] = req.memoryRequirements.size;
>  > +mem_size += mem_size_list[i];
>  > +continue;
>  > +}
>  > +
>  >  /* In case the implementation prefers/requires dedicated allocation
> */
>  >  use_ded_mem = ded_req.prefersDedicatedAllocation |
>  >  ded_req.requiresDedicatedAllocation;
>  > @@ -1648,6 +1668,29 @@ static int
>  alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
>  >  bind_info[i].memory = f->mem[i];
>  >  }
>  >
>  > +if (p->use_one_memory) {
>  > +memory_requirements.size = mem_size;
>  > +
>  > +/* Allocate memory */
>  > +if ((err = alloc_mem(ctx, &memory_requirements,
>  > +f->tiling == VK_IMAGE_TILING_LINEAR ?
>  > +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT :
>  > +VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
>  > +(void *)(((uint8_t *)alloc_pnext)),
>  > +&f->flags, &f->mem[0])))
>  > +return err;
>  > +
>  > +f->size[0] = memory_requirements.size;
>  > +
>  > +for (int i = 0; i < planes; i++) {
>  > +bind_info[i].sType  =
>  VK_STRUCTURE_TYPE_BIND_IMAGE_MEMORY_INFO;
>  > +bind_info[i].image  = f->img[i];
>  > +bind_info[i].memory = f->mem[0];
>  > +bind_info[i].memoryOffset = i == 0 ? 0 : 
>  > mem_size_list[i-1];
>  > +f->offset[i] = bind_info[i].memoryOffset;
>  > +}
>  > +}
>  > +
>  >  /* Bind the allocated memory to the images */
>  >  ret = vk->BindImageMemory2(hwctx->act_dev, planes, bind_info);
>  >  if (ret != VK_SUCCESS) {
>  > @@ -2924,7 +2967,8 @@ static int

Re: [FFmpeg-devel] [PATCH 7/7] libavutil/hwcontext_vulkan: specify the modifier to create VKImage

[Chen, Wenbin]

> 9 Nov 2021, 10:18 by wenbin.c...@intel.com:
> 
> > When vulkan image exports to drm, the tilling need to be
> > VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to
> create vulkan
> > image using this format.
> >
> > Now the following command line works:
> >
> > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -
> hwaccel_output_format \
> > vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan,
> \
> > scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v
> h264_vaapi output.264
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_vulkan.c | 76 +-
> --
> >  libavutil/hwcontext_vulkan.h |  5 +++
> >  2 files changed, 75 insertions(+), 6 deletions(-)
> >
> > diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> > index 29ade94b7f..e252c2177e 100644
> > --- a/libavutil/hwcontext_vulkan.c
> > +++ b/libavutil/hwcontext_vulkan.c
> > @@ -1919,6 +1919,7 @@ static void
> try_export_flags(AVHWFramesContext *hwfc,
> >  AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx;
> >  VulkanDevicePriv *p = hwfc->device_ctx->internal->priv;
> >  FFVulkanFunctions *vk = &p->vkfn;
> > +const int has_modifiers = hwctx->tiling ==
> VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT;
> >  VkExternalImageFormatProperties eprops = {
> >  .sType =
> VK_STRUCTURE_TYPE_EXTERNAL_IMAGE_FORMAT_PROPERTIES_KHR,
> >  };
> > @@ -1926,9 +1927,18 @@ static void
> try_export_flags(AVHWFramesContext *hwfc,
> >  .sType = VK_STRUCTURE_TYPE_IMAGE_FORMAT_PROPERTIES_2,
> >  .pNext = &eprops,
> >  };
> > +VkPhysicalDeviceImageDrmFormatModifierInfoEXT phy_dev_mod_info
> = {
> > +.sType =
> VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER
> _INFO_EXT,
> > +.pNext = NULL,
> > +.pQueueFamilyIndices   = p->qfs,
> > +.queueFamilyIndexCount = p->num_qfs,
> > +.sharingMode   = p->num_qfs > 1 ?
> VK_SHARING_MODE_CONCURRENT :
> > +  
> > VK_SHARING_MODE_EXCLUSIVE,
> > +};
> >  VkPhysicalDeviceExternalImageFormatInfo enext = {
> >  .sType =
> VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_IMAGE_FORMAT_INFO,
> >  .handleType = exp,
> > +.pNext = has_modifiers ? &phy_dev_mod_info : NULL,
> >  };
> >  VkPhysicalDeviceImageFormatInfo2 pinfo = {
> >  .sType  =
> VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_FORMAT_INFO_2,
> > @@ -1940,11 +1950,15 @@ static void
> try_export_flags(AVHWFramesContext *hwfc,
> >  .flags  = VK_IMAGE_CREATE_ALIAS_BIT,
> >  };
> >
> > -ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx-
> >phys_dev,
> > -  &pinfo, &props);
> > -if (ret == VK_SUCCESS) {
> > -*iexp |= exp;
> > -*comp_handle_types |=
> eprops.externalMemoryProperties.compatibleHandleTypes;
> > +for (int i = 0; i < (has_modifiers ? hwctx->modifier_count : 1); i++) {
> > +if (has_modifiers && hwctx->modifier_count)
> > +phy_dev_mod_info.drmFormatModifier = hwctx->modifiers[i];
> > +ret = vk->GetPhysicalDeviceImageFormatProperties2(dev_hwctx-
> >phys_dev,
> > +&pinfo, &props);
> > +if (ret == VK_SUCCESS) {
> > +*iexp |= exp;
> > +*comp_handle_types |=
> eprops.externalMemoryProperties.compatibleHandleTypes;
> > +}
> >  }
> >  }
> >
> > @@ -2007,6 +2021,7 @@ fail:
> >  static void vulkan_frames_uninit(AVHWFramesContext *hwfc)
> >  {
> >  VulkanFramesPriv *fp = hwfc->internal->priv;
> > +AVVulkanFramesContext *hwctx = hwfc->hwctx;
> >
> >  free_exec_ctx(hwfc, &fp->conv_ctx);
> >  free_exec_ctx(hwfc, &fp->upload_ctx);
> > @@ -2021,11 +2036,60 @@ static int
> vulkan_frames_init(AVHWFramesContext *hwfc)
> >  VulkanFramesPriv *fp = hwfc->internal->priv;
> >  AVVulkanDeviceContext *dev_hwctx = hwfc->device_ctx->hwctx;
> >  VulkanDevicePriv *p = hwfc->device_ctx->internal->priv;
> > +const int has_modifiers = !!(p->extensions &
> FF_VK_EXT_DRM_MODIFIER_FLAGS);
> >
> >  /* Default pool flags */
> > -hwctx->tiling = hwctx->tiling ? hwctx->tiling : p->use_linear_images ?
> > +hwctx->tiling = hwctx->tiling ? hwctx->tiling : has_modifiers ?
> > +VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT : p-
> >use_linear_images ?
> >  VK_IMAGE_TILING_LINEAR : VK_IMAGE_TILING_OPTIMAL;
> >
> > +/* get the supported modifier */
> > +if (has_modifiers) {
> > +const VkFormat *fmt = av_vkfmt_from_pixfmt(hwfc->sw_format);
> > +FFVulkanFunctions *vk = &p->vkfn;
> > +VkDrmFormatModifierPropertiesEXT
> mod_props[MAX_VULKAN_MODIFIERS];
> > +
> > +VkDrmFormatModifierPropertiesListEXT mod_props_list = {
> > +.sType =
> VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT,
> > +.pNext = NULL,
> > +.drmFormatModifierCount = 0,
> > +.pDrmFormatModifierProperties = NU

Re: [FFmpeg-devel] [PATCH 4/7] libavutil/hwcontext_vulkan: Allocate vkFrame in one memory

[Chen, Wenbin]

> 9 Nov 2021, 10:18 by wenbin.c...@intel.com:
> 
> > The vaapi can import external frame, but the planes of the external
> > frames should be in the same drm object. I add a new function to
> > allocate vkFrame in one memory and vulkan device will choose a way
> > to allocate memory according to one_memory flag.
> > A new variable is added to AVVKFrame to store the offset of each plane.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >  libavutil/hwcontext_vulkan.c | 46
> +++-
> >  libavutil/hwcontext_vulkan.h |  1 +
> >  2 files changed, 46 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> > index ccf3e58f49..f7878ed9c3 100644
> > --- a/libavutil/hwcontext_vulkan.c
> > +++ b/libavutil/hwcontext_vulkan.c
> > @@ -1600,6 +1600,9 @@ static int alloc_bind_mem(AVHWFramesContext
> *hwfc, AVVkFrame *f,
> >  FFVulkanFunctions *vk = &p->vkfn;
> >  const int planes = av_pix_fmt_count_planes(hwfc->sw_format);
> >  VkBindImageMemoryInfo bind_info[AV_NUM_DATA_POINTERS] = { { 0 } };
> > +VkMemoryRequirements memory_requirements = { 0 };
> > +int mem_size = 0;
> > +int mem_size_list[AV_NUM_DATA_POINTERS] = { 0 };
> >
> >  AVVulkanDeviceContext *hwctx = ctx->hwctx;
> >
> > @@ -1627,6 +1630,23 @@ static int
> alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
> >  req.memoryRequirements.size = FFALIGN(req.memoryRequirements.size,
> >  p->props.properties.limits.minMemoryMapAlignment);
> >
> > +if (p->use_one_memory) {
> > +if (ded_req.prefersDedicatedAllocation |
> ded_req.requiresDedicatedAllocation) {
> > +av_log(hwfc, AV_LOG_ERROR, "Cannot use dedicated allocation
> for intel vaapi\n");
> > +return AVERROR(EINVAL);
> > +}
> >
> 
> We don't set the flag unless the driver tells us to, so if the
> driver asks us to use dedicated memory when it can't handle such
> images, shouldn't the driver just not set this flag?

I check the dedicatedAllocation flag because I don't know if vaapi driver 
support importing dedicated memory.
Actually I am not sure if I need to check this flag for vaapi. I can remove it.

> 
> 
> > +if (memory_requirements.size == 0) {
> > +memory_requirements = req.memoryRequirements;
> > +} else if (memory_requirements.memoryTypeBits !=
> req.memoryRequirements.memoryTypeBits) {
> > +av_log(hwfc, AV_LOG_ERROR, "the param for each planes are
> not the same\n");
> > +return AVERROR(EINVAL);
> > +}
> > +
> > +mem_size_list[i] = req.memoryRequirements.size;
> > +mem_size += mem_size_list[i];
> > +continue;
> > +}
> > +
> >  /* In case the implementation prefers/requires dedicated allocation */
> >  use_ded_mem = ded_req.prefersDedicatedAllocation |
> >  ded_req.requiresDedicatedAllocation;
> > @@ -1648,6 +1668,29 @@ static int
> alloc_bind_mem(AVHWFramesContext *hwfc, AVVkFrame *f,
> >  bind_info[i].memory = f->mem[i];
> >  }
> >
> > +if (p->use_one_memory) {
> > +memory_requirements.size = mem_size;
> > +
> > +/* Allocate memory */
> > +if ((err = alloc_mem(ctx, &memory_requirements,
> > +f->tiling == VK_IMAGE_TILING_LINEAR ?
> > +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT :
> > +VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
> > +(void *)(((uint8_t *)alloc_pnext)),
> > +&f->flags, &f->mem[0])))
> > +return err;
> > +
> > +f->size[0] = memory_requirements.size;
> > +
> > +for (int i = 0; i < planes; i++) {
> > +bind_info[i].sType  =
> VK_STRUCTURE_TYPE_BIND_IMAGE_MEMORY_INFO;
> > +bind_info[i].image  = f->img[i];
> > +bind_info[i].memory = f->mem[0];
> > +bind_info[i].memoryOffset = i == 0 ? 0 : mem_size_list[i-1];
> > +f->offset[i] = bind_info[i].memoryOffset;
> > +}
> > +}
> > +
> >  /* Bind the allocated memory to the images */
> >  ret = vk->BindImageMemory2(hwctx->act_dev, planes, bind_info);
> >  if (ret != VK_SUCCESS) {
> > @@ -2924,7 +2967,8 @@ static int
> vulkan_map_to_drm(AVHWFramesContext *hwfc, AVFrame *dst,
> >  continue;
> >
> >  vk->GetImageSubresourceLayout(hwctx->act_dev, f->img[i], &sub,
> &layout);
> > -drm_desc->layers[i].planes[0].offset   = layout.offset;
> > +drm_desc->layers[i].planes[0].offset   = p->use_one_memory ?
> > +f->offset[i] : 
> > layout.offset;
> >  drm_desc->layers[i].planes[0].pitch= layout.rowPitch;
> >  }
> >
> > diff --git a/libavutil/hwcontext_vulkan.h b/libavutil/hwcontext_vulkan.h
> > index 9264f70dbf..efb602ef27 100644
> > --- a/libavutil/hwcontext_vulkan.h
> > +++ b/libavutil/hwcontext_vulkan.h
> > @@ -189,6 +189,7 

Re: [FFmpeg-devel] [PATCH 1/4] libavutil/hwcontext_d3d11va: Add nb_surfaces to AVD3D11VAFramesContext

[Chen, Wenbin]

> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> > Chen, Wenbin
> > Sent: Wednesday, November 10, 2021 4:03 AM
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH 1/4] libavutil/hwcontext_d3d11va:
> > Add nb_surfaces to AVD3D11VAFramesContext
> >
> > > Adding nb_surfaces in AVD3D11VAFramesContext in the end of the
> > structure
> > > to support flexible size of this arrays and align to
> > > AVDXVA2FramesContext and AVVAAPIFramesContext.
> 
> There is no flexibility in pool size for D3D11 frames contexts. The
> surface count is always identical to initial_pool_size. There's no
> point in exposing it. nb_surfaces could even be removed here.
> 
> Also, this change doesn't align with AVVAAPIFramesContext.
> In fact not even AVDXVA2FramesContext aligns with AVVAAPIFramesContext.
> Both have an nb_surfaces field, but they have different semantics.
> 
> The corresponding field to AVVAAPIFramesContext->nb_surfaces is
> DXVA2FramesContext->nb_surfaces_used, not AVDXVA2FramesContext-
> >nb_surfaces.
> 
> Kind regards,
> softworkz
> 
> 
Yes, the nb_surface for d3d11 seems useless, I can use init_pool_size instead.
The nb_surface in dxva works as the same as init_pool_size, and I can use 
init_pool_size instead as well. The original code is good I don't need to 
change it.
Since I found the real cause for segmentation fault, I will resubmit the patches

Thanks
Wenbin
> 
> 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix a bug when malloc handle_pairs_internal

[Chen, Wenbin]

> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> > Chen, Wenbin
> > Sent: Wednesday, November 10, 2021 4:03 AM
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix
> > a bug when malloc handle_pairs_internal
> >
> > > This commandline cause core dumped:
> > > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> > > -hwaccel_output_format vaapi -i input.264 \
> > > -vf "hwmap=derive_device=qsv,format=qsv" \
> > > -c:v h264_qsv output.264
> > >
> > > reason: We use nb_surfaces to assign surface to
> > handle_pairs_internal
> > > but handle_pairs_internal is alloced with the size of
> > init_pool_size.
> > > This lead to access to illegal address.
> > >
> > > Now change it to use nb_surfaces to allocate handle_pairs_internal
> > and the
> 
> I'm not sure about whether this is right.
> 
> When we look at the top of the qsv_frames_derive_to function that you
> are changing, there is this:
> 
> 
> if (src_ctx->initial_pool_size == 0) {
> av_log(dst_ctx, AV_LOG_ERROR, "Only fixed-size pools can be "
> "mapped to QSV frames.\n");
> return AVERROR(EINVAL);
> }
> 
> It's because QSV doesn't support dynamic pool sizes.
> 
> When we look at the vaapi_pool_alloc function in hwcontext_vaapi.c, we
> can see that:
> 
>   when  initial_pool_size is > 0, the pool cannot grow beyond this value,
>   so nb_surfaces cannot be > initial_pool_size
> 
> So I'm wondering what could have caused the segfault? Which values did
> you have there for nb_surfaces and initial_pool_size?
> 
> 
> > > core dumped error is unseen. Also change D3D11VA to use nb_surfaces
> > > to align to VAAPI and DXVA2.
> 
> Those changes are unrelated to fixing the issue with VAAPI.
> (besides that I don't think these are needed at all)
> 
> Kind regards,
> softworkz

You are right. The real cause is that vaapi_decode_make_config() is called 
twice.
The init_pool_size is changed on the second call. I will resubmit patch to fix 
this

Thanks
Wenbin
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 6/7] libavutil/hwcontext_vulkan: fix a sem_wait bug when export drm

[Chen, Wenbin]

> 9 Nov 2021, 10:18 by wenbin.c...@intel.com:
> 
> > sem_sig_val is wrongly assigned to pWaitSemaphoreValues when export
> drm. Now fix
> > it.
> >
> > Signed-off-by: Wenbin Chen <> wenbin.c...@intel.com> >
> >
> 
> Thanks for spotting this, I fixed that in my patchset and updated branch.
> frame->sem_value is safe to use for waiting, as it's only updated after
> the command buffer is successfully submitted.
> 
Ok, Got it.
 
Thanks
Wenbin
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 4/4] libavutil/hwcontext_opencl: fix a bug for mapping qsv frame to opencl

[Chen, Wenbin]

> From: nyanmisaka 
> 
> mfxHDLPair was added to qsv, so modify qsv->opencl map function as well.
> Now the following commandline works:
> 
> ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 \
> -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -
> filter_hw_device ocl \
> -hwaccel qsv -hwaccel_output_format qsv -hwaccel_device qs -c:v h264_qsv
> \
> -i input.264 -vf
> "hwmap=derive_device=opencl,format=opencl,avgblur_opencl, \
> hwmap=derive_device=qsv:reverse=1:extra_hw_frames=32,format=qsv" \
> -c:v h264_qsv output.264
> 
> Signed-off-by: nyanmisaka 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_opencl.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_opencl.c b/libavutil/hwcontext_opencl.c
> index 26a3a24593..4b6e74ff6f 100644
> --- a/libavutil/hwcontext_opencl.c
> +++ b/libavutil/hwcontext_opencl.c
> @@ -2249,7 +2249,8 @@ static int
> opencl_map_from_qsv(AVHWFramesContext *dst_fc, AVFrame *dst,
>  #if CONFIG_LIBMFX
>  if (src->format == AV_PIX_FMT_QSV) {
>  mfxFrameSurface1 *mfx_surface = (mfxFrameSurface1*)src->data[3];
> -va_surface = *(VASurfaceID*)mfx_surface->Data.MemId;
> +mfxHDLPair *pair = (mfxHDLPair*)mfx_surface->Data.MemId;
> +va_surface = *(VASurfaceID*)pair->first;
>  } else
>  #endif
>  if (src->format == AV_PIX_FMT_VAAPI) {
> --
> 2.25.1

ping
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 3/4] libavutil/hwcontext_qsv: fix a bug for mapping vaapi frame to qsv

[Chen, Wenbin]

> From: nyanmisaka 
> 
> The data stored in data[3] in VAAPI AVFrame is VASurfaceID while
> the data stored in pair->first is the pointer of VASurfaceID, so
> we need to do cast to make following commandline works:
> 
> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> -hwaccel_output_format vaapi -i input.264 \
> -vf "hwmap=derive_device=qsv,format=qsv" -c:v h264_qsv output.264
> 
> Signed-off-by: nyanmisaka 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_qsv.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> index 5a285fd25b..8075c27862 100644
> --- a/libavutil/hwcontext_qsv.c
> +++ b/libavutil/hwcontext_qsv.c
> @@ -1219,7 +1219,7 @@ static int qsv_map_to(AVHWFramesContext
> *dst_ctx,
>  case AV_PIX_FMT_VAAPI:
>  {
>  mfxHDLPair *pair = (mfxHDLPair*)hwctx->surfaces[i].Data.MemId;
> -if (pair->first == src->data[3]) {
> +if (*(VASurfaceID*)pair->first == (VASurfaceID)src->data[3]) {
>  index = i;
>  break;
>  }
> --
> 2.25.1

ping
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix a bug when malloc handle_pairs_internal

[Chen, Wenbin]

> This commandline cause core dumped:
> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> -hwaccel_output_format vaapi -i input.264 \
> -vf "hwmap=derive_device=qsv,format=qsv" \
> -c:v h264_qsv output.264
> 
> reason: We use nb_surfaces to assign surface to handle_pairs_internal
> but handle_pairs_internal is alloced with the size of init_pool_size.
> This lead to access to illegal address.
> 
> Now change it to use nb_surfaces to allocate handle_pairs_internal and the
> core dumped error is unseen. Also change D3D11VA to use nb_surfaces
> to align to VAAPI and DXVA2.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_qsv.c | 13 ++---
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> index c18747f7eb..5a285fd25b 100644
> --- a/libavutil/hwcontext_qsv.c
> +++ b/libavutil/hwcontext_qsv.c
> @@ -1123,8 +1123,7 @@ static int
> qsv_frames_derive_to(AVHWFramesContext *dst_ctx,
>  case AV_HWDEVICE_TYPE_VAAPI:
>  {
>  AVVAAPIFramesContext *src_hwctx = src_ctx->hwctx;
> -s->handle_pairs_internal = av_calloc(src_ctx->initial_pool_size,
> - 
> sizeof(*s->handle_pairs_internal));
> +s->handle_pairs_internal = av_calloc(src_hwctx->nb_surfaces,
> sizeof(*s->handle_pairs_internal));
>  if (!s->handle_pairs_internal)
>  return AVERROR(ENOMEM);
>  s->surfaces_internal = av_calloc(src_hwctx->nb_surfaces,
> @@ -1146,15 +1145,15 @@ static int
> qsv_frames_derive_to(AVHWFramesContext *dst_ctx,
>  case AV_HWDEVICE_TYPE_D3D11VA:
>  {
>  AVD3D11VAFramesContext *src_hwctx = src_ctx->hwctx;
> -s->handle_pairs_internal = av_calloc(src_ctx->initial_pool_size,
> +s->handle_pairs_internal = av_calloc(src_ctx->nb_surfaces,
>   
> sizeof(*s->handle_pairs_internal));
>  if (!s->handle_pairs_internal)
>  return AVERROR(ENOMEM);
> -s->surfaces_internal = av_calloc(src_ctx->initial_pool_size,
> +s->surfaces_internal = av_calloc(src_ctx->nb_surfaces,
>   sizeof(*s->surfaces_internal));
>  if (!s->surfaces_internal)
>  return AVERROR(ENOMEM);
> -for (i = 0; i < src_ctx->initial_pool_size; i++) {
> +for (i = 0; i < src_ctx->nb_surfaces; i++) {
>  qsv_init_surface(dst_ctx, &s->surfaces_internal[i]);
>  s->handle_pairs_internal[i].first = (mfxMemId)src_hwctx-
> >texture_infos[i].texture;
>  if (src_hwctx->BindFlags & D3D11_BIND_RENDER_TARGET) {
> @@ -1164,7 +1163,7 @@ static int
> qsv_frames_derive_to(AVHWFramesContext *dst_ctx,
>  }
>  s->surfaces_internal[i].Data.MemId = (mfxMemId)&s-
> >handle_pairs_internal[i];
>  }
> -dst_hwctx->nb_surfaces = src_ctx->initial_pool_size;
> +dst_hwctx->nb_surfaces = src_ctx->nb_surfaces;
>  if (src_hwctx->BindFlags & D3D11_BIND_RENDER_TARGET) {
>  dst_hwctx->frame_type |=
> MFX_MEMTYPE_VIDEO_MEMORY_PROCESSOR_TARGET;
>  } else {
> @@ -1177,7 +1176,7 @@ static int
> qsv_frames_derive_to(AVHWFramesContext *dst_ctx,
>  case AV_HWDEVICE_TYPE_DXVA2:
>  {
>  AVDXVA2FramesContext *src_hwctx = src_ctx->hwctx;
> -s->handle_pairs_internal = av_calloc(src_ctx->initial_pool_size,
> +s->handle_pairs_internal = av_calloc(src_ctx->nb_surfaces,
>   
> sizeof(*s->handle_pairs_internal));
>  if (!s->handle_pairs_internal)
>  return AVERROR(ENOMEM);
> --
> 2.25.1

ping
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/4] libavutil/hwcontext_d3d11va: Add nb_surfaces to AVD3D11VAFramesContext

[Chen, Wenbin]

> Adding nb_surfaces in AVD3D11VAFramesContext in the end of the structure
> to support flexible size of this arrays and align to
> AVDXVA2FramesContext and AVVAAPIFramesContext.
> 
> Signed-off-by Wenbin Chen 
> ---
>  libavutil/hwcontext_d3d11va.c | 3 +--
>  libavutil/hwcontext_d3d11va.h | 2 ++
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/libavutil/hwcontext_d3d11va.c b/libavutil/hwcontext_d3d11va.c
> index 8ab96bad25..086e7b9daa 100644
> --- a/libavutil/hwcontext_d3d11va.c
> +++ b/libavutil/hwcontext_d3d11va.c
> @@ -72,7 +72,6 @@ static av_cold void load_functions(void)
>  }
> 
>  typedef struct D3D11VAFramesContext {
> -int nb_surfaces;
>  int nb_surfaces_used;
> 
>  DXGI_FORMAT format;
> @@ -287,7 +286,7 @@ static int d3d11va_frames_init(AVHWFramesContext
> *ctx)
>  hwctx->texture_infos = av_calloc(ctx->initial_pool_size, sizeof(*hwctx-
> >texture_infos));
>  if (!hwctx->texture_infos)
>  return AVERROR(ENOMEM);
> -s->nb_surfaces = ctx->initial_pool_size;
> +hwctx->nb_surfaces = ctx->initial_pool_size;
> 
>  ctx->internal->pool_internal =
> av_buffer_pool_init2(sizeof(AVD3D11FrameDescriptor),
>  ctx, 
> d3d11va_pool_alloc, NULL);
> diff --git a/libavutil/hwcontext_d3d11va.h b/libavutil/hwcontext_d3d11va.h
> index 77d2d72f1b..b0df470190 100644
> --- a/libavutil/hwcontext_d3d11va.h
> +++ b/libavutil/hwcontext_d3d11va.h
> @@ -173,6 +173,8 @@ typedef struct AVD3D11VAFramesContext {
>   * This field is ignored/invalid if a user-allocated texture is provided.
>  */
>  AVD3D11FrameDescriptor *texture_infos;
> +
> +int nb_surfaces;
>  } AVD3D11VAFramesContext;
> 
>  #endif /* AVUTIL_HWCONTEXT_D3D11VA_H */
> --
> 2.25.1

ping
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v3] libavfilter: add a gblur_vulkan filter

[Chen, Wenbin]

> 9 Sept 2021, 07:44 by jianhua...@intel.com:
> 
> > This commit adds a powerful and customizable gblur Vulkan filter,
> > which provides a maximum 127x127 kernel size of Gaussian Filter.
> > The size could be adjusted by requirements on quality or performance.
> >
> > The following command is on how to apply gblur_vulkan filter:
> >
> > ffmpeg -init_hw_device vulkan=vul:0 -filter_hw_device vul -i input.264
> > -vf
> hwupload=extra_hw_frames=16,gblur_vulkan,hwdownload,format=yuv420p
> > output.264
> >
> > Signed-off-by: Wu Jianhua 
> > ---
> >  configure |   1 +
> >  libavfilter/Makefile  |   1 +
> >  libavfilter/allfilters.c  |   1 +
> >  libavfilter/vf_gblur_vulkan.c | 511
> ++
> >  4 files changed, 514 insertions(+)
> >  create mode 100644 libavfilter/vf_gblur_vulkan.c
> >
> > diff --git a/configure b/configure
> > index af410a9d11..4b9a0d8e07 100755
> > --- a/configure
> > +++ b/configure
> > @@ -3601,6 +3601,7 @@ freezedetect_filter_select="scene_sad"
> >  frei0r_filter_deps="frei0r libdl"
> >  frei0r_src_filter_deps="frei0r libdl"
> >  fspp_filter_deps="gpl"
> > +gblur_vulkan_filter_deps="vulkan_lib libglslang"
> >  histeq_filter_deps="gpl"
> >  hqdn3d_filter_deps="gpl"
> >  interlace_filter_deps="gpl"
> > diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> > index af957a5ac0..5f74e33599 100644
> > --- a/libavfilter/Makefile
> > +++ b/libavfilter/Makefile
> > @@ -286,6 +286,7 @@ OBJS-$(CONFIG_FREEZEFRAMES_FILTER)   +=
> vf_freezeframes.o
> >  OBJS-$(CONFIG_FREI0R_FILTER) += vf_frei0r.o
> >  OBJS-$(CONFIG_FSPP_FILTER)   += vf_fspp.o qp_table.o
> >  OBJS-$(CONFIG_GBLUR_FILTER)  += vf_gblur.o
> > +OBJS-$(CONFIG_GBLUR_VULKAN_FILTER)   += vf_gblur_vulkan.o
> vulkan.o
> >  OBJS-$(CONFIG_GEQ_FILTER)+= vf_geq.o
> >  OBJS-$(CONFIG_GRADFUN_FILTER)+= vf_gradfun.o
> >  OBJS-$(CONFIG_GRAPHMONITOR_FILTER)   += f_graphmonitor.o
> > diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> > index 0c6b2347c8..b5576de3af 100644
> > --- a/libavfilter/allfilters.c
> > +++ b/libavfilter/allfilters.c
> > @@ -271,6 +271,7 @@ extern const AVFilter ff_vf_freezeframes;
> >  extern const AVFilter ff_vf_frei0r;
> >  extern const AVFilter ff_vf_fspp;
> >  extern const AVFilter ff_vf_gblur;
> > +extern const AVFilter ff_vf_gblur_vulkan;
> >  extern const AVFilter ff_vf_geq;
> >  extern const AVFilter ff_vf_gradfun;
> >  extern const AVFilter ff_vf_graphmonitor;
> > diff --git a/libavfilter/vf_gblur_vulkan.c b/libavfilter/vf_gblur_vulkan.c
> > new file mode 100644
> > index 00..5c072f8971
> > --- /dev/null
> > +++ b/libavfilter/vf_gblur_vulkan.c
> > @@ -0,0 +1,511 @@
> > +/*
> > + * copyright (c) 2021 Wu Jianhua 
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
> USA
> > + */
> > +
> > +#include "libavutil/random_seed.h"
> > +#include "libavutil/opt.h"
> > +#include "vulkan.h"
> > +#include "internal.h"
> > +
> > +#define CGS 32
> > +#define GBLUR_MAX_KERNEL_SIZE 127
> > +
> > +typedef struct GBlurVulkanContext {
> > +VulkanFilterContext vkctx;
> > +FFVkExecContext *exec;
> > +VulkanPipeline *pl_hor;
> > +VulkanPipeline *pl_ver;
> > +FFVkBuffer params_buf_hor;
> > +FFVkBuffer params_buf_ver;
> > +
> > +VkDescriptorImageInfo input_images[3];
> > +VkDescriptorImageInfo tmp_images[3];
> > +VkDescriptorImageInfo output_images[3];
> > +VkDescriptorBufferInfo params_desc_hor;
> > +VkDescriptorBufferInfo params_desc_ver;
> > +
> > +int initialized;
> > +int size;
> > +int planes;
> > +int kernel_size;
> > +float sigma;
> > +float sigmaV;
> > +AVFrame *tmpframe;
> > +} GBlurVulkanContext;
> > +
> > +static const char gblur_horizontal[] = {
> > +C(0, void gblur(const ivec2 pos, const int index)  
> > )
> > +C(0, { 
> > )
> > +C(1, vec4 sum = texture(input_image[index], pos) *
> kernel[0];  )
> > +C(0,   
> 

Re: [FFmpeg-devel] [PATCH 3/3] libavcodec/vaapi_encode: Add async_depth to vaapi_encoder to increase performance

[Chen, Wenbin]

> Add async_depth to increase encoder's performance. Reuse encode_fifo as
> async buffer. Encoder puts all reordered frame to HW and then check
> fifo size. If fifo < async_depth and the top frame is not ready, it will
> return AVERROR(EAGAIN) to require more frames.
> 
> 1080p transcoding (no B frames) with -async_depth=4 can increase 20%
> performance on my environment.
> The async increases performance but also introduces frame delay.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/vaapi_encode.c | 20 +++-
>  libavcodec/vaapi_encode.h | 12 ++--
>  2 files changed, 25 insertions(+), 7 deletions(-)
> 
> diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> index db0ae136a1..616fb7c089 100644
> --- a/libavcodec/vaapi_encode.c
> +++ b/libavcodec/vaapi_encode.c
> @@ -1158,7 +1158,8 @@ static int
> vaapi_encode_send_frame(AVCodecContext *avctx, AVFrame *frame)
>  if (ctx->input_order == ctx->decode_delay)
>  ctx->dts_pts_diff = pic->pts - ctx->first_pts;
>  if (ctx->output_delay > 0)
> -ctx->ts_ring[ctx->input_order % (3 * ctx->output_delay)] = 
> pic->pts;
> +ctx->ts_ring[ctx->input_order %
> +(3 * ctx->output_delay + ctx->async_depth)] = 
> pic->pts;
> 
>  pic->display_order = ctx->input_order;
>  ++ctx->input_order;
> @@ -1212,7 +1213,8 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
>  return AVERROR(EAGAIN);
>  }
> 
> -while (av_fifo_size(ctx->encode_fifo) <= MAX_PICTURE_REFERENCES *
> sizeof(VAAPIEncodePicture *)) {
> +while (av_fifo_size(ctx->encode_fifo) <
> +MAX_ASYNC_DEPTH * sizeof(VAAPIEncodePicture *)) {
>  pic = NULL;
>  err = vaapi_encode_pick_next(avctx, &pic);
>  if (err < 0)
> @@ -1234,6 +1236,14 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
>  if (!av_fifo_size(ctx->encode_fifo))
>  return err;
> 
> +if (av_fifo_size(ctx->encode_fifo) < ctx->async_depth *
> sizeof(VAAPIEncodePicture *) &&
> +!ctx->end_of_stream) {
> +av_fifo_generic_peek(ctx->encode_fifo, &pic, sizeof(pic), NULL);
> +err = vaapi_encode_wait(avctx, pic, 0);
> +if (err < 0)
> +return err;
> +}
> +
>  av_fifo_generic_read(ctx->encode_fifo, &pic, sizeof(pic), NULL);
>  ctx->encode_order = pic->encode_order + 1;
> 
> @@ -1252,7 +1262,7 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
>  pkt->dts = ctx->ts_ring[pic->encode_order] - ctx->dts_pts_diff;
>  } else {
>  pkt->dts = ctx->ts_ring[(pic->encode_order - ctx->decode_delay) %
> -(3 * ctx->output_delay)];
> +(3 * ctx->output_delay + ctx->async_depth)];
>  }
>  av_log(avctx, AV_LOG_DEBUG, "Output packet: pts %"PRId64"
> dts %"PRId64".\n",
> pkt->pts, pkt->dts);
> @@ -2566,8 +2576,8 @@ av_cold int ff_vaapi_encode_init(AVCodecContext
> *avctx)
>  }
>  }
> 
> -ctx->encode_fifo = av_fifo_alloc((MAX_PICTURE_REFERENCES + 1) *
> -  sizeof(VAAPIEncodePicture *));
> +ctx->encode_fifo = av_fifo_alloc(MAX_ASYNC_DEPTH *
> + sizeof(VAAPIEncodePicture *));
>  if (!ctx->encode_fifo)
>  return AVERROR(ENOMEM);
> 
> diff --git a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h
> index 89fe8de466..1bf5d7c337 100644
> --- a/libavcodec/vaapi_encode.h
> +++ b/libavcodec/vaapi_encode.h
> @@ -48,6 +48,7 @@ enum {
>  MAX_TILE_ROWS  = 22,
>  // A.4.1: table A.6 allows at most 20 tile columns for any level.
>  MAX_TILE_COLS  = 20,
> +MAX_ASYNC_DEPTH= 64,
>  };
> 
>  extern const AVCodecHWConfigInternal *const
> ff_vaapi_encode_hw_configs[];
> @@ -298,7 +299,8 @@ typedef struct VAAPIEncodeContext {
>  // Timestamp handling.
>  int64_t first_pts;
>  int64_t dts_pts_diff;
> -int64_t ts_ring[MAX_REORDER_DELAY * 3];
> +int64_t ts_ring[MAX_REORDER_DELAY * 3 +
> +MAX_ASYNC_DEPTH];
> 
>  // Slice structure.
>  int slice_block_rows;
> @@ -348,6 +350,8 @@ typedef struct VAAPIEncodeContext {
>  AVFrame *frame;
> 
>  AVFifoBuffer *encode_fifo;
> +
> +int async_depth;
>  } VAAPIEncodeContext;
> 
>  enum {
> @@ -458,7 +462,11 @@ int ff_vaapi_encode_close(AVCodecContext *avctx);
>  { "b_depth", \
>"Maximum B-frame reference depth", \
>OFFSET(common.desired_b_depth), AV_OPT_TYPE_INT, \
> -  { .i64 = 1 }, 1, INT_MAX, FLAGS }
> +  { .i64 = 1 }, 1, INT_MAX, FLAGS }, \
> +{ "async_depth", "Maximum processing parallelism. " \
> +  "Increase this to improve single channel performance", \
> +  OFFSET(common.async_depth), AV_OPT_TYPE_INT, \
> +  { .i64 = 4 }, 0, MAX_ASYNC_DE

Re: [FFmpeg-devel] [PATCH 2/3] libavcodec/vaapi_encode: Add new API adaption to vaapi_encode

[Chen, Wenbin]

> Add vaSyncBuffer to VAAPI encoder. Old version API vaSyncSurface wait
> surface to complete. When surface is used for multiple operation, it
> wait all operation to finish. vaSyncBuffer only wait one channel to
> finish.
> 
> Add wait param to vaapi_encode_wait() to prepare for the async_depth
> option. "wait=1" means wait until operation ready. "wait=0" means
> query operation's status. If ready return 0, if still in progress
> return EAGAIN.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/vaapi_encode.c | 47 +--
>  1 file changed, 40 insertions(+), 7 deletions(-)
> 
> diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> index 5927849233..db0ae136a1 100644
> --- a/libavcodec/vaapi_encode.c
> +++ b/libavcodec/vaapi_encode.c
> @@ -134,7 +134,8 @@ static int
> vaapi_encode_make_misc_param_buffer(AVCodecContext *avctx,
>  }
> 
>  static int vaapi_encode_wait(AVCodecContext *avctx,
> - VAAPIEncodePicture *pic)
> + VAAPIEncodePicture *pic,
> + uint8_t wait)
>  {
>  VAAPIEncodeContext *ctx = avctx->priv_data;
>  VAStatus vas;
> @@ -150,11 +151,43 @@ static int vaapi_encode_wait(AVCodecContext
> *avctx,
> "(input surface %#x).\n", pic->display_order,
> pic->encode_order, pic->input_surface);
> 
> -vas = vaSyncSurface(ctx->hwctx->display, pic->input_surface);
> -if (vas != VA_STATUS_SUCCESS) {
> -av_log(avctx, AV_LOG_ERROR, "Failed to sync to picture completion: "
> -   "%d (%s).\n", vas, vaErrorStr(vas));
> +#if VA_CHECK_VERSION(1, 9, 0)
> +// Try vaSyncBuffer.
> +vas = vaSyncBuffer(ctx->hwctx->display,
> +   pic->output_buffer,
> +   wait ? VA_TIMEOUT_INFINITE : 0);
> +if (vas == VA_STATUS_ERROR_TIMEDOUT) {
> +return AVERROR(EAGAIN);
> +} else if (vas != VA_STATUS_SUCCESS && vas !=
> VA_STATUS_ERROR_UNIMPLEMENTED) {
> +av_log(avctx, AV_LOG_ERROR, "Failed to sync to output buffer
> completion: "
> +"%d (%s).\n", vas, vaErrorStr(vas));
>  return AVERROR(EIO);
> +} else if (vas == VA_STATUS_ERROR_UNIMPLEMENTED)
> +// If vaSyncBuffer is not implemented, try old version API.
> +#endif
> +{
> +if (!wait) {
> +VASurfaceStatus surface_status;
> +vas = vaQuerySurfaceStatus(ctx->hwctx->display,
> +pic->input_surface,
> +&surface_status);
> +if (vas == VA_STATUS_SUCCESS &&
> +surface_status != VASurfaceReady &&
> +surface_status != VASurfaceSkipped) {
> +return AVERROR(EAGAIN);
> +} else if (vas != VA_STATUS_SUCCESS) {
> +av_log(avctx, AV_LOG_ERROR, "Failed to query surface status: 
> "
> +"%d (%s).\n", vas, vaErrorStr(vas));
> +return AVERROR(EIO);
> +}
> +} else {
> +vas = vaSyncSurface(ctx->hwctx->display, pic->input_surface);
> +if (vas != VA_STATUS_SUCCESS) {
> +av_log(avctx, AV_LOG_ERROR, "Failed to sync to picture
> completion: "
> +"%d (%s).\n", vas, vaErrorStr(vas));
> +return AVERROR(EIO);
> +}
> +}
>  }
> 
>  // Input is definitely finished with now.
> @@ -633,7 +666,7 @@ static int vaapi_encode_output(AVCodecContext
> *avctx,
>  uint8_t *ptr;
>  int err;
> 
> -err = vaapi_encode_wait(avctx, pic);
> +err = vaapi_encode_wait(avctx, pic, 1);
>  if (err < 0)
>  return err;
> 
> @@ -695,7 +728,7 @@ fail:
>  static int vaapi_encode_discard(AVCodecContext *avctx,
>  VAAPIEncodePicture *pic)
>  {
> -vaapi_encode_wait(avctx, pic);
> +vaapi_encode_wait(avctx, pic, 1);
> 
>  if (pic->output_buffer_ref) {
>  av_log(avctx, AV_LOG_DEBUG, "Discard output for pic "
> --
> 2.25.1

ping
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/3] libavcodec/vaapi_encode: Change the way to call async to increase performance

[Chen, Wenbin]

> Fix: #7706. After commit 5fdcf85bbffe7451c2, vaapi encoder's performance
> decrease. The reason is that vaRenderPicture() and vaSyncSurface() are
> called at the same time (vaRenderPicture() always followed by a
> vaSyncSurface()). When we encode stream with B frames, we need buffer to
> reorder frames, so we can send serveral frames to HW at once to increase
> performance. Now I changed them to be called in a
> asynchronous way, which will make better use of hardware.
> 1080p transcoding increases about 17% fps on my environment.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavcodec/vaapi_encode.c | 41 ---
>  libavcodec/vaapi_encode.h |  3 +++
>  2 files changed, 33 insertions(+), 11 deletions(-)
> 
> diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> index ec054ae701..5927849233 100644
> --- a/libavcodec/vaapi_encode.c
> +++ b/libavcodec/vaapi_encode.c
> @@ -951,8 +951,10 @@ static int vaapi_encode_pick_next(AVCodecContext
> *avctx,
>  if (!pic && ctx->end_of_stream) {
>  --b_counter;
>  pic = ctx->pic_end;
> -if (pic->encode_issued)
> +if (pic->encode_complete)
>  return AVERROR_EOF;
> +else if (pic->encode_issued)
> +return AVERROR(EAGAIN);
>  }
> 
>  if (!pic) {
> @@ -1177,20 +1179,31 @@ int
> ff_vaapi_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt)
>  return AVERROR(EAGAIN);
>  }
> 
> -pic = NULL;
> -err = vaapi_encode_pick_next(avctx, &pic);
> -if (err < 0)
> -return err;
> -av_assert0(pic);
> +while (av_fifo_size(ctx->encode_fifo) <= MAX_PICTURE_REFERENCES *
> sizeof(VAAPIEncodePicture *)) {
> +pic = NULL;
> +err = vaapi_encode_pick_next(avctx, &pic);
> +if (err < 0)
> +break;
> +av_assert0(pic);
> 
> -pic->encode_order = ctx->encode_order++;
> +pic->encode_order = ctx->encode_order +
> +(av_fifo_size(ctx->encode_fifo) / 
> sizeof(VAAPIEncodePicture
> *));
> 
> -err = vaapi_encode_issue(avctx, pic);
> -if (err < 0) {
> -av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> -return err;
> +err = vaapi_encode_issue(avctx, pic);
> +if (err < 0) {
> +av_log(avctx, AV_LOG_ERROR, "Encode failed: %d.\n", err);
> +return err;
> +}
> +
> +av_fifo_generic_write(ctx->encode_fifo, &pic, sizeof(pic), NULL);
>  }
> 
> +if (!av_fifo_size(ctx->encode_fifo))
> +return err;
> +
> +av_fifo_generic_read(ctx->encode_fifo, &pic, sizeof(pic), NULL);
> +ctx->encode_order = pic->encode_order + 1;
> +
>  err = vaapi_encode_output(avctx, pic, pkt);
>  if (err < 0) {
>  av_log(avctx, AV_LOG_ERROR, "Output failed: %d.\n", err);
> @@ -2520,6 +2533,11 @@ av_cold int
> ff_vaapi_encode_init(AVCodecContext *avctx)
>  }
>  }
> 
> +ctx->encode_fifo = av_fifo_alloc((MAX_PICTURE_REFERENCES + 1) *
> +  sizeof(VAAPIEncodePicture *));
> +if (!ctx->encode_fifo)
> +return AVERROR(ENOMEM);
> +
>  return 0;
> 
>  fail:
> @@ -2552,6 +2570,7 @@ av_cold int
> ff_vaapi_encode_close(AVCodecContext *avctx)
> 
>  av_freep(&ctx->codec_sequence_params);
>  av_freep(&ctx->codec_picture_params);
> +av_fifo_freep(&ctx->encode_fifo);
> 
>  av_buffer_unref(&ctx->recon_frames_ref);
>  av_buffer_unref(&ctx->input_frames_ref);
> diff --git a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h
> index b41604a883..89fe8de466 100644
> --- a/libavcodec/vaapi_encode.h
> +++ b/libavcodec/vaapi_encode.h
> @@ -29,6 +29,7 @@
> 
>  #include "libavutil/hwcontext.h"
>  #include "libavutil/hwcontext_vaapi.h"
> +#include "libavutil/fifo.h"
> 
>  #include "avcodec.h"
>  #include "hwconfig.h"
> @@ -345,6 +346,8 @@ typedef struct VAAPIEncodeContext {
>  int roi_warned;
> 
>  AVFrame *frame;
> +
> +AVFifoBuffer *encode_fifo;
>  } VAAPIEncodeContext;
> 
>  enum {
> --
> 2.25.1

ping

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V2] libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

[Chen, Wenbin]

> Command below failed.
> ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128
> -init_hw_device qsv=qs@va -hwaccel qsv -hwaccel_device qs
> -filter_hw_device va -c:v h264_qsv
> -i 1080P.264 -vf "hwmap,format=vaapi" -c:v h264_vaapi output.264
> 
> Cause: Assign pair->first directly to data[3] in vaapi frame.
> pair->first is *VASurfaceID while data[3] in vaapi frame is
> VASurfaceID. I fix this line of code. Now the command above works.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_qsv.c | 6 +-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> index d431e71eab..b01236889e 100644
> --- a/libavutil/hwcontext_qsv.c
> +++ b/libavutil/hwcontext_qsv.c
> @@ -781,7 +781,11 @@ static int qsv_map_from(AVHWFramesContext *ctx,
>  case AV_HWDEVICE_TYPE_VAAPI:
>  {
>  mfxHDLPair *pair = (mfxHDLPair*)surf->Data.MemId;
> -child_data = pair->first;
> +/* pair->first is *VASurfaceID while data[3] in vaapi frame is
> VASurfaceID, so
> + * we need this casting for vaapi.
> + * Add intptr_t to force cast from VASurfaceID(uint) type to
> pointer(long) type
> + * to avoid compile warning */
> +child_data = (uint8_t*)(intptr_t)*(VASurfaceID*)pair->first;
>  break;
>  }
>  #endif
> --
> 2.25.1

Ping

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH V2] libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

[Chen, Wenbin]

> Command below failed.
> ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128
> -init_hw_device qsv=qs@va -hwaccel qsv -hwaccel_device qs
> -filter_hw_device va -c:v h264_qsv
> -i 1080P.264 -vf "hwmap,format=vaapi" -c:v h264_vaapi output.264
> 
> Cause: Assign pair->first directly to data[3] in vaapi frame.
> pair->first is *VASurfaceID while data[3] in vaapi frame is
> VASurfaceID. I fix this line of code. Now the command above works.
> 
> Signed-off-by: Wenbin Chen 
> ---
>  libavutil/hwcontext_qsv.c | 6 +-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> index d431e71eab..b01236889e 100644
> --- a/libavutil/hwcontext_qsv.c
> +++ b/libavutil/hwcontext_qsv.c
> @@ -781,7 +781,11 @@ static int qsv_map_from(AVHWFramesContext *ctx,
>  case AV_HWDEVICE_TYPE_VAAPI:
>  {
>  mfxHDLPair *pair = (mfxHDLPair*)surf->Data.MemId;
> -child_data = pair->first;
> +/* pair->first is *VASurfaceID while data[3] in vaapi frame is
> VASurfaceID, so
> + * we need this casting for vaapi.
> + * Add intptr_t to force cast from VASurfaceID(uint) type to
> pointer(long) type
> + * to avoid compile warning */
> +child_data = (uint8_t*)(intptr_t)*(VASurfaceID*)pair->first;
>  break;
>  }
>  #endif
> --
> 2.25.1

ping


___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

[Chen, Wenbin]

> On Fri, 2021-09-10 at 02:19 +0000, Chen, Wenbin wrote:
> > > -Original Message-
> > > From: ffmpeg-devel  On Behalf Of
> > > James Almer
> > > Sent: Thursday, September 9, 2021 8:48 PM
> > > To: ffmpeg-devel@ffmpeg.org
> > > Subject: Re: [FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: fix a bug
> for
> > > mapping qsv frame to vaapi
> > >
> > > On 9/9/2021 6:13 AM, Wenbin Chen wrote:
> > > > Command below failed.
> > > > ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128
> > > > -init_hw_device qsv=qs@va -hwaccel qsv -hwaccel_device qs
> > > > -filter_hw_device va -c:v h264_qsv
> > > > -i 1080P.264 -vf "hwmap,format=vaapi" -c:v h264_vaapi output.264
> > > >
> > > > Cause: Assign pair->first directly to data[3] in vaapi frame.
> > > > pair->first is *VASurfaceID while data[3] in vaapi frame is
> > > > VASurfaceID. I fix this line of code. Now the command above works.
> > > >
> > > > Signed-off-by: Wenbin Chen 
> > > > ---
> > > >   libavutil/hwcontext_qsv.c | 2 +-
> > > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> > > > index d431e71eab..6539cae619 100644
> > > > --- a/libavutil/hwcontext_qsv.c
> > > > +++ b/libavutil/hwcontext_qsv.c
> > > > @@ -781,7 +781,7 @@ static int qsv_map_from(AVHWFramesContext
> *ctx,
> > > >   case AV_HWDEVICE_TYPE_VAAPI:
> > > >   {
> > > >   mfxHDLPair *pair = (mfxHDLPair*)surf->Data.MemId;
> > > > -child_data = pair->first;
> > > > +child_data = (uint8_t*)(intptr_t)*(VASurfaceID*)pair->first;
> > >
> > > You can probably remove the intptr_t casting.
> >
> > If intptr_t is removed, it will report compile warning
> > "cast to pointer from integer of different size"
> 
> How about to add a comment here? If so, others can understand why this
> casting
> is needed.
> 
> Thanks
> Haihao

Ok, I will send path V2

> 
> 
> >
> > >
> > > Also, shouldn't this same fix be done for all three child device types
> > > used in this function? Which for that matter, child_data seems to be set
> > > for a d3d11va child device, but then never used.
> >
> > Dxva and d3d11 frames store the pointer to surface while vaapi frames
> store
> > surface,
> > so this part of code for dxva and d3d11va should be correct.
> >
> > D3d11 and dxva share dxva codec, the child_data is used in this part of
> code.
> > >
> > > >   break;
> > > >   }
> > > >   #endif
> > > >
> > >
> > > ___
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> >
> > ___
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

[Chen, Wenbin]

> -Original Message-
> From: ffmpeg-devel  On Behalf Of
> James Almer
> Sent: Thursday, September 9, 2021 8:48 PM
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: fix a bug for
> mapping qsv frame to vaapi
> 
> On 9/9/2021 6:13 AM, Wenbin Chen wrote:
> > Command below failed.
> > ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128
> > -init_hw_device qsv=qs@va -hwaccel qsv -hwaccel_device qs
> > -filter_hw_device va -c:v h264_qsv
> > -i 1080P.264 -vf "hwmap,format=vaapi" -c:v h264_vaapi output.264
> >
> > Cause: Assign pair->first directly to data[3] in vaapi frame.
> > pair->first is *VASurfaceID while data[3] in vaapi frame is
> > VASurfaceID. I fix this line of code. Now the command above works.
> >
> > Signed-off-by: Wenbin Chen 
> > ---
> >   libavutil/hwcontext_qsv.c | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> > index d431e71eab..6539cae619 100644
> > --- a/libavutil/hwcontext_qsv.c
> > +++ b/libavutil/hwcontext_qsv.c
> > @@ -781,7 +781,7 @@ static int qsv_map_from(AVHWFramesContext *ctx,
> >   case AV_HWDEVICE_TYPE_VAAPI:
> >   {
> >   mfxHDLPair *pair = (mfxHDLPair*)surf->Data.MemId;
> > -child_data = pair->first;
> > +child_data = (uint8_t*)(intptr_t)*(VASurfaceID*)pair->first;
> 
> You can probably remove the intptr_t casting.

If intptr_t is removed, it will report compile warning
"cast to pointer from integer of different size"
 
> 
> Also, shouldn't this same fix be done for all three child device types
> used in this function? Which for that matter, child_data seems to be set
> for a d3d11va child device, but then never used.

Dxva and d3d11 frames store the pointer to surface while vaapi frames store 
surface, 
so this part of code for dxva and d3d11va should be correct.

D3d11 and dxva share dxva codec, the child_data is used in this part of code.
> 
> >   break;
> >   }
> >   #endif
> >
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avutils/hwcontext: When deriving a hwdevice, search for existing device in both directions

[Chen, Wenbin]

> -Original Message-
> From: ffmpeg-devel  On Behalf Of Soft
> Works
> Sent: Wednesday, September 8, 2021 2:25 PM
> To: FFmpeg development discussions and patches  de...@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v2] avutils/hwcontext: When deriving a
> hwdevice, search for existing device in both directions
> 
> 
> 
> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> > Chen, Wenbin
> > Sent: Wednesday, 8 September 2021 07:58
> > To: FFmpeg development discussions and patches  > de...@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH v2] avutils/hwcontext: When
> > deriving a hwdevice, search for existing device in both directions
> >
> > Hi softworkz:
> >
> > Any updates on this patch?
> > I want to submit pathes to qsv, but they depends on this patch
> >
> > Thanks
> > Wenbin
> 
> Hi Wenbin,
> 
> Haihao has discovered a regression but I haven't looked into it
> yet. Will do within the next few days.
> 
> softworkz

Got it
Thank you

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avutils/hwcontext: When deriving a hwdevice, search for existing device in both directions

[Chen, Wenbin]

Hi softworkz:

Any updates on this patch?
I want to submit pathes to qsv, but they depends on this patch

Thanks
Wenbin
> -Original Message-
> From: ffmpeg-devel  On Behalf Of Soft
> Works
> Sent: Thursday, August 19, 2021 3:52 PM
> To: FFmpeg development discussions and patches  de...@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v2] avutils/hwcontext: When deriving a
> hwdevice, search for existing device in both directions
> 
> 
> 
> > -Original Message-
> > From: ffmpeg-devel  On Behalf Of
> Xiang,
> > Haihao
> > Sent: Thursday, 19 August 2021 09:37
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH v2] avutils/hwcontext: When deriving a
> > hwdevice, search for existing device in both directions
> >
> > On Fri, 2021-08-13 at 06:29 +, Xiang, Haihao wrote:
> > > On Tue, 2021-08-10 at 09:52 +, Soft Works wrote:
> > > > The test /libavutil/tests/hwdevice checks that when deriving a device
> > > > from a source device and then deriving back to the type of the source
> > > > device, the result is matching the original source device, i.e. the
> > > > derivation mechanism doesn't create a new device in this case.
> > > >
> 
> [..]
> 
> > >
> > > video:143kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
> > muxing
> > > overhead: unknown
> > > corrupted size vs. prev_size in fastbins
> > > Aborted
> > >
> >
> > Hi Softworks,
> >
> > +child_device->internal->derived_devices[ctx->type] =
> > av_buffer_create((uint8_t*)ctx, sizeof(*ctx), 0, ctx, 0);
> >
> > The above change introduces a new AVBufferRef for ctx. The first
> AVBufferRef
> > for
> > ctx is created when function av_hwdevice_ctx_alloc is called. So there are
> > two
> > different AVBufferRefs referring to the same ctx, then ctx will be double-
> > freed
> >
> > The change below is a bit ugly, but it may fix this double-free issue.
> >
> > +static void qsv_ctx_free(void *opaque, uint8_t *ctx)
> > +{
> > +// Do nothing here
> > +// ctx is freed in hwdevice_ctx_free
> > +}
> > +
> >  static int qsv_device_create(AVHWDeviceContext *ctx, const char *device,
> >   AVDictionary *opts, int flags)
> >  {
> > @@ -1271,7 +1277,7 @@ static int qsv_device_create(AVHWDeviceContext
> *ctx,
> > const
> > char *device,
> >  ret = qsv_device_derive_from_child(ctx, impl, child_device, 0);
> >  if (ret >= 0) {
> >  ctx->internal->source_device = av_buffer_ref(priv-
> > >child_device_ctx);
> > -child_device->internal->derived_devices[ctx->type] =
> > av_buffer_create((uint8_t*)ctx, sizeof(*ctx), 0, ctx, 0);
> > +child_device->internal->derived_devices[ctx->type] =
> > av_buffer_create((uint8_t*)ctx, sizeof(*ctx), qsv_ctx_free, ctx, 0);
> >  if (!child_device->internal->derived_devices[ctx->type]) {
> >  return AVERROR(ENOMEM);
> >  }
> 
> Hi Haihao,
> 
> sorry for getting back late and thanks for your suggested fix.
> I'll apply it and submit a new patch version.
> 
> Best regards,
> softworkz
> 
> 
> 
> 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [V2 08/10] libavutil/hwcontext_vulkan: fix wrong offset of plane

[Chen, Wenbin]

> -Original Message-
> From: Lynne 
> Sent: Wednesday, September 1, 2021 7:33 PM
> To: Chen, Wenbin 
> Subject: RE: [FFmpeg-devel] [V2 08/10] libavutil/hwcontext_vulkan: fix wrong
> offset of plane
> 
> 1 Sept 2021, 03:40 by wenbin.c...@intel.com:
> 
> >
> >
> >> -Original Message-
> >> From: ffmpeg-devel  On Behalf Of
> >> Lynne
> >> Sent: Tuesday, August 31, 2021 7:42 PM
> >> To: FFmpeg development discussions and patches  >> de...@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [V2 08/10] libavutil/hwcontext_vulkan: fix
> wrong
> >> offset of plane
> >>
> >> 31 Aug 2021, 09:24 by wenbin.c...@intel.com:
> >>
> >> > According to spec, if we use VkBindImagePlaneMemoryInfo to bind
> image
> >> > we mush create image with disjoint flag.
> >> > The offset in subresourcelayout is relative to the base address of
> >> > the plane, but the offset in drm is relative to the drm objectis so
> >> > I think this offset should be 0.
> >> > Also, when I import vaapi frame to vulkan I got broken frame, and
> >> > setting plane_data->offset to 0 makes command works.
> >> >
> >> > Signed-off-by: Wenbin Chen 
> >> >
> >>
> >> I don't quite get the point of this. Is there some situation where
> >> clients may want to change the modifier of the image they're allocating,
> >> like maybe use a different modifier when allocating images that would
> >> be used as framebuffers to be presented?
> >>
> >
> > Maybe you reply to the wrong patch? I assume you want to reply to the 9th
> patch.
> > " specify the modifier to create VKImage".
> > I tried on the latest intel driver. The log recommend to use linear or
> drmModifier to create image.
> > Also if I use optimal to create image, the export drm modifier is unknown.
> >
> 
> Linear is almost never optimal. Could you post a link to the recommendation?
> The DRM modifier is known for optimal, there's a Vulkan API to retrieve
> image modifiers.

This restriction is for VK_EXTERNAL_MEMORY_HANDLE_TYPE_DMA_BUF_BIT_EXT.

I get recommendation information from here.
https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/intel/vulkan/anv_formats.c#L1412
If I use optimal to create VkImage, the modifier I get from 
VkGetImageDrmFormatModifierPropertiesExt() 
is DRM_FORMAT_MOD_INVALID.
I tested this on Intel platform, and I see AMD has this restriction as well.
https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/amd/vulkan/radv_formats.c#L1530
 

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [V2 03/10] libavfilter/vulkan: Fix the way to use sem

[Chen, Wenbin]

> -Original Message-
> From: ffmpeg-devel  On Behalf Of
> Lynne
> Sent: Tuesday, August 31, 2021 7:26 PM
> To: FFmpeg development discussions and patches  de...@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [V2 03/10] libavfilter/vulkan: Fix the way to use
> sem
> 
> 31 Aug 2021, 09:24 by wenbin.c...@intel.com:
> 
> > We chould set waitSem and signalSem differently. Current ffmpeg-vulkan
> > uses the same sem to set waitSem and signalSem and it doesn't work on
> > latest intel-vulkan-driver. The commit:
> > a193060221c4df123e26a562949cae5df3e73cde on mesa causes this
> problem.
> > This commit add code to resets the signalSem. This will reset waitSem
> > too on current ffmpeg-vulkan. Now set waitSem and signalSem separetely.
> >
> > Now the following command can run on the latest mesa on intel platform:
> > ffmpeg -v verbose -init_hw_device vulkan=vul:0,linear_images=1 -
> filter_hw_device vul
> > -i input1080p.264 -vf
> "hwupload=extra_hw_frames=16,scale_vulkan=1920:1080,
> > hwdownload,format=yuv420p" -f rawvideo output.yuv
> >
> > Signed-off-by: Wenbin Chen 
> >
> 
> Sorry, I'm working on a better way which uses timeline semaphores.
> Would've been nice to get pinged on IRC with what your plans were
> beforehand.

It is good you have a better way to do this. Look forward to your fix. :D

> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [V2 08/10] libavutil/hwcontext_vulkan: fix wrong offset of plane

[Chen, Wenbin]

> -Original Message-
> From: ffmpeg-devel  On Behalf Of
> Lynne
> Sent: Tuesday, August 31, 2021 7:42 PM
> To: FFmpeg development discussions and patches  de...@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [V2 08/10] libavutil/hwcontext_vulkan: fix wrong
> offset of plane
> 
> 31 Aug 2021, 09:24 by wenbin.c...@intel.com:
> 
> > According to spec, if we use VkBindImagePlaneMemoryInfo to bind image
> > we mush create image with disjoint flag.
> > The offset in subresourcelayout is relative to the base address of
> > the plane, but the offset in drm is relative to the drm objectis so
> > I think this offset should be 0.
> > Also, when I import vaapi frame to vulkan I got broken frame, and
> > setting plane_data->offset to 0 makes command works.
> >
> > Signed-off-by: Wenbin Chen 
> >
> 
> I don't quite get the point of this. Is there some situation where
> clients may want to change the modifier of the image they're allocating,
> like maybe use a different modifier when allocating images that would
> be used as framebuffers to be presented?

Maybe you reply to the wrong patch? I assume you want to reply to the 9th patch.
" specify the modifier to create VKImage". 
I tried on the latest intel driver. The log recommend to use linear or 
drmModifier to create image.
Also if I use optimal to create image, the export drm modifier is unknown.

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".