Re: [FFmpeg-devel] [PATCH] avcodec/hevcdec: check that the local context list exists before dereferencing it
On Wed, Feb 10, 2021 at 6:57 PM Andreas Rheinhardt <
andreas.rheinha...@gmail.com> wrote:
> James Almer:
> > Since the decoder is not flagged as init cleanup capable,
> hevc_decode_free()
> > is being called manually if the hevc_decode_extradata() call fails at
> the end
> > of hevc_decode_init().
> > In a frame threading scenario, however, if AVCodec->init() returns an
> error,
> > ff_frame_thread_free() will be called regardless of the above flag being
> set
> > or not, resulting in hevc_decode_free() being called a second time for
> the
> > same context.
> >
> > Solve this by ensuring pointers are not dereferenced if they are NULL,
> and
> > setting the decoder as init cleanup capable.
> >
> > Fixes ticket #9099.
> >
> > Signed-off-by: James Almer
> > ---
> > Maybe ff_frame_thread_free() should not call AVCodec->close() for thread
> contexts
> > where AVCodec->init() failed and FF_CODEC_CAP_INIT_CLEANUP is not set?
> >
>
> Fixing this has been on my to-do list. (The situation is even worse than
> you describe it: It is possible that AVCodec->close is called on an
> AVCodecContext whose private_data couldn't be allocated.)
>
So how should proceed? Apply this patch and fix other issues after it?
>
> > libavcodec/hevcdec.c | 9 -
> > 1 file changed, 4 insertions(+), 5 deletions(-)
> >
> > diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c
> > index 92eb888033..898dac8cbb 100644
> > --- a/libavcodec/hevcdec.c
> > +++ b/libavcodec/hevcdec.c
> > @@ -3417,6 +3417,7 @@ static av_cold int hevc_decode_free(AVCodecContext
> *avctx)
> > av_freep(>sh.offset);
> > av_freep(>sh.size);
> >
> > +if (s->HEVClcList && s->sList) {
> > for (i = 1; i < s->threads_number; i++) {
> > HEVCLocalContext *lc = s->HEVClcList[i];
> > if (lc) {
> > @@ -3424,9 +3425,8 @@ static av_cold int hevc_decode_free(AVCodecContext
> *avctx)
> > av_freep(>sList[i]);
> > }
> > }
> > -if (s->HEVClc == s->HEVClcList[0])
> > -s->HEVClc = NULL;
> > -av_freep(>HEVClcList[0]);
> > +}
> > +av_freep(>HEVClc);
> > av_freep(>HEVClcList);
> > av_freep(>sList);
> >
> > @@ -3622,7 +3622,6 @@ static av_cold int hevc_decode_init(AVCodecContext
> *avctx)
> > if (avctx->extradata_size > 0 && avctx->extradata) {
> > ret = hevc_decode_extradata(s, avctx->extradata,
> avctx->extradata_size, 1);
> > if (ret < 0) {
> > -hevc_decode_free(avctx);
> > return ret;
> > }
> > }
> > @@ -3673,7 +3672,7 @@ AVCodec ff_hevc_decoder = {
> > .capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY |
> > AV_CODEC_CAP_SLICE_THREADS |
> AV_CODEC_CAP_FRAME_THREADS,
> > .caps_internal = FF_CODEC_CAP_INIT_THREADSAFE |
> FF_CODEC_CAP_EXPORTS_CROPPING |
> > - FF_CODEC_CAP_ALLOCATE_PROGRESS,
> > + FF_CODEC_CAP_ALLOCATE_PROGRESS |
> FF_CODEC_CAP_INIT_CLEANUP,
> > .profiles = NULL_IF_CONFIG_SMALL(ff_hevc_profiles),
> > .hw_configs= (const AVCodecHWConfigInternal *const []) {
> > #if CONFIG_HEVC_DXVA2_HWACCEL
> >
>
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] libsvtav1: Add logical_processors option
On Wed, Feb 10, 2021 at 11:08 PM Mark Thompson wrote:
> On 10/02/2021 17:16, Christopher Degawa wrote:
> > From: Christopher Degawa
> >
> > Equivalent to the --lp option for SvtAv1EncApp, and is the only way
> > to control how much cpu power svt-av1 uses for now
> >
> > Not using thread_count as it would be preferable to reserve that until
> > svt-av1 properly implements a threads option
>
> Then what does it actually do? The description below about the number of
> logical threads to run sounds pretty much exactly like what the threads
> option does.
>
>
Threads option is limited to filters that do filtering by calling
execute(), internal threading code of libavfilter.
> > 0 == getconf _NPROCESSORS_ONLN
> >
> > Signed-off-by: Christopher Degawa
> > ---
> > libavcodec/libsvtav1.c | 7 +++
> > 1 file changed, 7 insertions(+)
> >
> > diff --git a/libavcodec/libsvtav1.c b/libavcodec/libsvtav1.c
> > index eb6043bcac..2296735f25 100644
> > --- a/libavcodec/libsvtav1.c
> > +++ b/libavcodec/libsvtav1.c
> > @@ -71,6 +71,8 @@ typedef struct SvtContext {
> >
> > int tile_columns;
> > int tile_rows;
> > +
> > +unsigned logical_processors;
> > } SvtContext;
> >
> > static const struct {
> > @@ -218,6 +220,8 @@ static int
> config_enc_params(EbSvtAv1EncConfiguration *param,
> > param->tile_columns = svt_enc->tile_columns;
> > param->tile_rows= svt_enc->tile_rows;
> >
> > +param->logical_processors = svt_enc->logical_processors;
> > +
> > return 0;
> > }
> >
> > @@ -533,6 +537,9 @@ static const AVOption options[] = {
> > { "tile_columns", "Log2 of number of tile columns to use",
> OFFSET(tile_columns), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 4, VE},
> > { "tile_rows", "Log2 of number of tile rows to use",
> OFFSET(tile_rows), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 6, VE},
> >
> > +{ "logical_processors", "Number of logical processors to run the
> encoder on, threads are managed by the OS scheduler",
> OFFSET(logical_processors),
> > + AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, VE },
> > +
> > {NULL},
> > };
> >
> >
>
> - Mark
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] libsvtav1: Add logical_processors option
Based on my limited understanding of the code, it's limiting the usage using pthread_setaffinity_np and CPU_SET on Linux to limit the process to certain CPUs, but it also has a default and max of the return of `sysconf(_SC_NPROCESSORS_ONLN)`. According to Hassene Tmar of SVT-AV1, it is a "target core count that SVT would do a best effort to achieve" and "--lp 8 might still produce 50 threads". It does not actually limit how many threads are deployed. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH V2 08/10] libavutil: add side data AVDnnBoundingBox for dnn based detect/classify filters
On 10/02/2021 09:34, Guo, Yejun wrote:
Signed-off-by: Guo, Yejun
---
doc/APIchanges | 2 ++
libavutil/Makefile | 1 +
libavutil/dnn_bbox.h | 68
libavutil/frame.c| 1 +
libavutil/frame.h| 7 +
libavutil/version.h | 2 +-
6 files changed, 80 insertions(+), 1 deletion(-)
create mode 100644 libavutil/dnn_bbox.h
What is the intended consumer of this box information? (Is there some other
filter which will read these are do something with them, or some sort of user
program?)
If there is no use in ffmpeg outside libavfilter then the header should
probably be in libavfilter.
How tied is this to the DNN implementation, and hence the DNN name? If someone
made a standalone filter doing object detection by some other method, would it
make sense for them to reuse this structure?
diff --git a/libavutil/dnn_bbox.h b/libavutil/dnn_bbox.h
new file mode 100644
index 00..50899c4486
--- /dev/null
+++ b/libavutil/dnn_bbox.h
@@ -0,0 +1,68 @@
+/*
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef AVUTIL_DNN_BBOX_H
+#define AVUTIL_DNN_BBOX_H
+
+#include "rational.h"
+
+typedef struct AVDnnBoundingBox {
+/**
+ * Must be set to the size of this data structure (that is,
+ * sizeof(AVDnnBoundingBox)).
+ */
+uint32_t self_size;
+
+/**
+ * Object detection is usually applied to a smaller image that
+ * is scaled down from the original frame.
+ * width and height are attributes of the scaled image, in pixel.
+ */
+int model_input_width;
+int model_input_height;
Other than to interpret the distances below, what will the user do with this
information? (Alternatively: why not map the distances back onto the original
frame size?)
+
+/**
+ * Distance in pixels from the top edge of the scaled image to top
+ * and bottom, and from the left edge of the scaled image to left and
+ * right, defining the bounding box.
+ */
+int top;
+int left;
+int bottom;
+int right;
+
+/**
+ * Detect result
+ */
+int detect_label;
How does a user interpret this label? Is it from some known enum?
+AVRational detect_conf;
"conf"... idence? A longer name and a descriptive comment might help.
+
+/**
+ * At most 4 classifications based on the detected bounding box.
+ * For example, we can get max 4 different attributes with 4 different
+ * DNN models on one bounding box.
+ * classify_count is zero if no classification.
+ */
+#define AV_NUM_BBOX_CLASSIFY 4
+uint32_t classify_count;
+int classify_labels[AV_NUM_BBOX_CLASSIFY];
+AVRational classify_confs[AV_NUM_BBOX_CLASSIFY];
Same comment on these.
+} AVDnnBoundingBox;
+
+#endif
diff --git a/libavutil/frame.c b/libavutil/frame.c
index eab51b6a32..4308507827 100644
--- a/libavutil/frame.c
+++ b/libavutil/frame.c
@@ -852,6 +852,7 @@ const char *av_frame_side_data_name(enum
AVFrameSideDataType type)
case AV_FRAME_DATA_VIDEO_ENC_PARAMS:return "Video encoding
parameters";
case AV_FRAME_DATA_SEI_UNREGISTERED:return "H.26[45] User Data
Unregistered SEI message";
case AV_FRAME_DATA_FILM_GRAIN_PARAMS: return "Film grain
parameters";
+case AV_FRAME_DATA_DNN_BBOXES: return "DNN bounding
boxes";
}
return NULL;
}
diff --git a/libavutil/frame.h b/libavutil/frame.h
index 1aeafef6de..a4dcfd27c9 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -198,6 +198,13 @@ enum AVFrameSideDataType {
* Must be present for every frame which should have film grain applied.
*/
AV_FRAME_DATA_FILM_GRAIN_PARAMS,
+
+/**
+ * Bounding box generated by dnn based filters for object detection and
classification,
+ * the data is an array of AVDnnBoudingBox, the number of array element is
implied by
+ * AVFrameSideData.size / AVDnnBoudingBox.self_size.
+ */
+AV_FRAME_DATA_DNN_BBOXES,
};
enum AVActiveFormatDescription {
- Mark
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
Re: [FFmpeg-devel] [PATCH] libsvtav1: Add logical_processors option
On 10/02/2021 17:16, Christopher Degawa wrote:
From: Christopher Degawa
Equivalent to the --lp option for SvtAv1EncApp, and is the only way
to control how much cpu power svt-av1 uses for now
Not using thread_count as it would be preferable to reserve that until
svt-av1 properly implements a threads option
Then what does it actually do? The description below about the number of
logical threads to run sounds pretty much exactly like what the threads option
does.
0 == getconf _NPROCESSORS_ONLN
Signed-off-by: Christopher Degawa
---
libavcodec/libsvtav1.c | 7 +++
1 file changed, 7 insertions(+)
diff --git a/libavcodec/libsvtav1.c b/libavcodec/libsvtav1.c
index eb6043bcac..2296735f25 100644
--- a/libavcodec/libsvtav1.c
+++ b/libavcodec/libsvtav1.c
@@ -71,6 +71,8 @@ typedef struct SvtContext {
int tile_columns;
int tile_rows;
+
+unsigned logical_processors;
} SvtContext;
static const struct {
@@ -218,6 +220,8 @@ static int config_enc_params(EbSvtAv1EncConfiguration
*param,
param->tile_columns = svt_enc->tile_columns;
param->tile_rows= svt_enc->tile_rows;
+param->logical_processors = svt_enc->logical_processors;
+
return 0;
}
@@ -533,6 +537,9 @@ static const AVOption options[] = {
{ "tile_columns", "Log2 of number of tile columns to use",
OFFSET(tile_columns), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 4, VE},
{ "tile_rows", "Log2 of number of tile rows to use", OFFSET(tile_rows),
AV_OPT_TYPE_INT, {.i64 = 0}, 0, 6, VE},
+{ "logical_processors", "Number of logical processors to run the encoder on, threads are managed by the OS scheduler", OFFSET(logical_processors),
+ AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, VE },
+
{NULL},
};
- Mark
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] libavcodec/hevcdsp: port SIMD idct functions from 32-bit.
Hi Reimar, On Sat, 16 Jan 2021, Martin Storsjö wrote: +.macro idct_16x16 bitdepth +function ff_hevc_idct_16x16_\bitdepth\()_neon, export=1 +//r0 - coeffs +mov x15, lr + Binutils doesn't recognize "lr" as alias for x30 It didn’t have an issue in the Debian unstable VM? That seems like the kind of workaround where it would be better to leave a comment with more info, if you know what exactly is affected. Binutils 2.28 doesn't recognize "lr" while 2.30 does, it seems. FWIW, all the existing aarch64 assembly just uses "x30" to refer to this register, none of it uses "lr". Do you want to follow up on this patch? IIRC changing it to use "x30" instead of "lr" was the only blocker from my point of view (and the add_residual patch that goes on top of it was mostly fine as well)? // Martin ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] lavu/tx: support in-place FFT transforms
Feb 10, 2021, 18:15 by d...@lynne.ee: > This commit adds support for in-place FFT transforms. Since our > internal transforms were all in-place anyway, this only changes > the permutation on the input. > > Unfortunately, research papers were of no help here. All focused > on dry hardware implementations, where permutes are free, or on > software implementations where binary bloat is of no concern so > storing dozen times the transforms for each permutation and version > is not considered bad practice. > Still, for a pure C implementation, it's only around 28% slower > than the multi-megabyte FFTW3 in unaligned mode. > > Unlike a closed permutation like with PFA, split-radix FFT bit-reversals > contain multiple NOPs, multiple simple swaps, and a few chained swaps, > so regular single-loop single-state permute loops were not possible. > Instead, we filter out parts of the input indices which are redundant. > This allows for a single branch, and with some clever AVX512 asm, > could possibly be SIMD'd without refactoring. > > The inplace_idx array is guaranteed to never be larger than the > revtab array, and in practice only requires around log2(len) entries. > > The power-of-two MDCTs can be done in-place as well. And it's > possible to eliminate a copy in the compound MDCTs too, however > it'll be slower than doing them out of place, and we'd need to dirty > the input array. > > Patch attached. > Locally added APIchanges and lavu minor bump. And got rid of the unused set temporary variables when permuting. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH 1/3] avformat/flvdec: Treat high ts byte as unsigned
On Tue, Nov 24, 2020 at 12:42:02AM +0100, Michael Niedermayer wrote: > Fixes: left shift of 255 by 24 places cannot be represented in type 'int' > Fixes: > 27516/clusterfuzz-testcase-minimized-ffmpeg_dem_KUX_fuzzer-5152854660349952 > > Signed-off-by: Michael Niedermayer > --- > libavformat/flvdec.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) will apply patchset [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB The real ebay dictionary, page 2 "100% positive feedback" - "All either got their money back or didnt complain" "Best seller ever, very honest" - "Seller refunded buyer after failed scam" signature.asc Description: PGP signature ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avformat/movenc: remove unnecessary HDR metadata log messages
On Wed, Feb 10, 2021 at 12:43 PM Paul B Mahol wrote: > > lgtm Cheers, applied as 752f14ff942576dd8697320fb6ba055593a2de9d . Jan ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v3] avformat/concatdec: add support for setting input options
On Wed, Feb 10, 2021 at 8:33 PM Nicolas George wrote:
>
> Jan Ekström (12021-02-08):
> > This way protocol or format related options can be set for all
> > of the files opened during concatenation both globally as well
> > as per-file.
> > ---
> >
> > Changes from v2:
> >
> > 1. Added an example, although I had issues figuring out something useful
> >that is not a hack for something. Ended up doing a disablement of
> >advanced edit list functionality, since that can sometimes lead to
> >unwanted behavior.
> >
>
> >* First idea was to override the input format for a file without an
> > extension. For that, we have no AVOption it seems.
>
> Indeed, lavf does not accept it as an option. It needs to be added
> separately. Or we can make it an option.
>
> >* Then came the idea of utilizing the framerate option in the raw
> > h264 demuxer. That didn't work because apparently if there is a header
> > in there that probed/parsed frame rate field gets utilized.
>
> You could set the frame rate for a raw video stream.
>
Sure.
> >* Third idea was either the one I picked, or analyzeduration/probesize
> > for MPEG-TS. I opted for the mp4 case.
> >
> > 2. Quoted the : in documentation with @code{:}.
> > 3. Fixed a duplicate space in a log message.
> > ---
> >
> > doc/demuxers.texi | 24 ++
> > libavformat/concatdec.c | 69 -
> > 2 files changed, 92 insertions(+), 1 deletion(-)
> >
> > diff --git a/doc/demuxers.texi b/doc/demuxers.texi
> > index 3c15ab9eee..20601f9575 100644
> > --- a/doc/demuxers.texi
> > +++ b/doc/demuxers.texi
> > @@ -149,6 +149,14 @@ Metadata of the packets of the file. The specified
> > metadata will be set for
> > each file packet. You can specify this directive multiple times to add
> > multiple
> > metadata entries.
> >
> > +@item @code{input_options @var{key=value:key2=value2}}
>
> > +Input options passed on when reading a specific file, using a
> > @code{:}-separated
> > +list of key=value pairs.
>
> No! Why would you do that? It adds one level of escaping, we already
> have way too much of them.
Because:
1. we have the functionality for it
2. it matches the AVOption.
Please do not take it as if I did not have any logic behind my actions
whatsoever, which is what the exclamation mark looks like.
>
> input option key=value
>
> singular, and can be set as many times as necessary.
>
> > Requires @code{safe} to be non-positive.
>
> Zero would be safer.
>
> > Global options
> > +for all files can be set with the @code{input_options} demuxer option.
> > When using
> > +both options on the list of files as well as globally via the demuxer
> > option,
> > +the global ones get applied first and the file-specific options are then
> > applied
> > +on top of them.
>
> I am not sure it is the right order. Options added on the fly should
> take precedence over options written in a file, in general. But the
> global / specific distinction makes it less clear-cut.
>
You set the AVOption for option(s) to be applied for all files, and if
you want to only set some option for specific files you set them in
the list.
*You* requested the latter capability, I added it.
> Other opinions?
>
> > +
> > @item @code{stream}
> > Introduce a stream in the virtual file.
> > All subsequent stream-related directives apply to the last introduced
> > @@ -204,6 +212,10 @@ expressed in microseconds. The duration metadata is
> > only set if it is known
> > based on the concat file.
> > The default is 0.
> >
> > +@item input_options
> > +Input options to be passed on for all opened inputs using
>
> > a :-separated list of
>
> It should say "a dictionary", with a link to where the syntax of
> dictionaries is documented, but whoever added AV_OPT_TYPE_DICT neglected
> to add the corresponding documentation paragraph :(
>
Yes. This just matches the rest of the documentation strings etc for
such things.
Jan
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH]lavfi/vflip: Support Bayer vertical flip
Just apply this, it is very unlikely that there are faster ways. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v3] avformat/concatdec: add support for setting input options
Jan Ekström (12021-02-08):
> This way protocol or format related options can be set for all
> of the files opened during concatenation both globally as well
> as per-file.
> ---
>
> Changes from v2:
>
> 1. Added an example, although I had issues figuring out something useful
>that is not a hack for something. Ended up doing a disablement of
>advanced edit list functionality, since that can sometimes lead to
>unwanted behavior.
>
>* First idea was to override the input format for a file without an
> extension. For that, we have no AVOption it seems.
Indeed, lavf does not accept it as an option. It needs to be added
separately. Or we can make it an option.
>* Then came the idea of utilizing the framerate option in the raw
> h264 demuxer. That didn't work because apparently if there is a header
> in there that probed/parsed frame rate field gets utilized.
You could set the frame rate for a raw video stream.
>* Third idea was either the one I picked, or analyzeduration/probesize
> for MPEG-TS. I opted for the mp4 case.
>
> 2. Quoted the : in documentation with @code{:}.
> 3. Fixed a duplicate space in a log message.
> ---
>
> doc/demuxers.texi | 24 ++
> libavformat/concatdec.c | 69 -
> 2 files changed, 92 insertions(+), 1 deletion(-)
>
> diff --git a/doc/demuxers.texi b/doc/demuxers.texi
> index 3c15ab9eee..20601f9575 100644
> --- a/doc/demuxers.texi
> +++ b/doc/demuxers.texi
> @@ -149,6 +149,14 @@ Metadata of the packets of the file. The specified
> metadata will be set for
> each file packet. You can specify this directive multiple times to add
> multiple
> metadata entries.
>
> +@item @code{input_options @var{key=value:key2=value2}}
> +Input options passed on when reading a specific file, using a
> @code{:}-separated
> +list of key=value pairs.
No! Why would you do that? It adds one level of escaping, we already
have way too much of them.
input option key=value
singular, and can be set as many times as necessary.
> Requires @code{safe} to be non-positive.
Zero would be safer.
> Global options
> +for all files can be set with the @code{input_options} demuxer option. When
> using
> +both options on the list of files as well as globally via the demuxer option,
> +the global ones get applied first and the file-specific options are then
> applied
> +on top of them.
I am not sure it is the right order. Options added on the fly should
take precedence over options written in a file, in general. But the
global / specific distinction makes it less clear-cut.
Other opinions?
> +
> @item @code{stream}
> Introduce a stream in the virtual file.
> All subsequent stream-related directives apply to the last introduced
> @@ -204,6 +212,10 @@ expressed in microseconds. The duration metadata is only
> set if it is known
> based on the concat file.
> The default is 0.
>
> +@item input_options
> +Input options to be passed on for all opened inputs using
> a :-separated list of
It should say "a dictionary", with a link to where the syntax of
dictionaries is documented, but whoever added AV_OPT_TYPE_DICT neglected
to add the corresponding documentation paragraph :(
> +key=value pairs.
> +
> @end table
>
> @subsection Examples
> @@ -231,6 +243,18 @@ duration 20.0
>
> file subdir/file-2.wav
> @end example
> +
> +@item
> +Disabling advanced edit list capability for the first input file via
> +input_options:
> +@example
> +ffconcat version 1.0
> +
> +file file-1.mp4
> +input_options advanced_editlist=false
> +
> +file file-2.mp4
> +@end example
> @end itemize
>
> @section dash
> diff --git a/libavformat/concatdec.c b/libavformat/concatdec.c
> index 6d5b9914f9..89d75cedc6 100644
> --- a/libavformat/concatdec.c
> +++ b/libavformat/concatdec.c
> @@ -52,6 +52,7 @@ typedef struct {
> int64_t outpoint;
> AVDictionary *metadata;
> int nb_streams;
> +AVDictionary *input_options;
> } ConcatFile;
>
> typedef struct {
> @@ -66,6 +67,7 @@ typedef struct {
> ConcatMatchMode stream_match_mode;
> unsigned auto_convert;
> int segment_time_metadata;
> +AVDictionary *input_options;
> } ConcatContext;
>
> static int concat_probe(const AVProbeData *probe)
> @@ -329,6 +331,7 @@ static int open_file(AVFormatContext *avf, unsigned
> fileno)
> {
> ConcatContext *cat = avf->priv_data;
> ConcatFile *file = >files[fileno];
> +AVDictionary *options = NULL;
> int ret;
>
> if (cat->avf)
> @@ -344,12 +347,37 @@ static int open_file(AVFormatContext *avf, unsigned
> fileno)
> if ((ret = ff_copy_whiteblacklists(cat->avf, avf)) < 0)
> return ret;
>
> -if ((ret = avformat_open_input(>avf, file->url, NULL, NULL)) < 0 ||
> +// Apply global AVOptions first
> +if (cat->input_options &&
> +(ret = av_dict_copy(, cat->input_options, 0) < 0))
> +return ret;
> +
> +// then apply file-specific AVOptions
Re: [FFmpeg-devel] [PATCH 4/8] avformat/mov: Factor offset advancement out in probing
On Sat, Feb 06, 2021 at 11:07:35PM +0100, Paul B Mahol wrote:
> LGTM
will apply
{...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
Opposition brings concord. Out of discord comes the fairest harmony.
-- Heraclitus
signature.asc
Description: PGP signature
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH 2/8] avformat/mov: Support size = 1 and size = 0 special cases in probing
On Mon, Feb 08, 2021 at 02:25:50PM +0100, Michael Niedermayer wrote:
> On Sat, Feb 06, 2021 at 11:33:38AM -0800, Chad Fraleigh wrote:
> > On 2/6/2021 9:22 AM, Michael Niedermayer wrote:
> > > Signed-off-by: Michael Niedermayer
> > > ---
> > > libavformat/mov.c | 5 +
> > > 1 file changed, 5 insertions(+)
> > >
> > > diff --git a/libavformat/mov.c b/libavformat/mov.c
> > > index 9406e42f49..70f76caff5 100644
> > > --- a/libavformat/mov.c
> > > +++ b/libavformat/mov.c
> > > @@ -7113,6 +7113,11 @@ static int mov_probe(const AVProbeData *p)
> > > if ((offset + 8) > (unsigned int)p->buf_size)
> > > break;
> > > size = AV_RB32(p->buf + offset);
> > > +if (size == 1 && offset + 16 > (unsigned int)p->buf_size) {
> > > +size = AV_RB64(p->buf+offset + 8);
> >
> > Just curious, what happens when size == 1 and the buffer is too small? Is
> > leaving it as a size of 1 still valid, or should it be handled as a format
> > error (e.g. abort the loop)?
>
> The buffer must have a minimum padding of AVPROBE_PADDING_SIZE
> so the buffer cannot be too small. This extra padding requirement is
> there for exactly cases like this, otherwise alot more checks would be
> needed in many probe functions
will apply
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
There will always be a question for which you do not know the correct answer.
signature.asc
Description: PGP signature
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avcodec/hevcdec: check that the local context list exists before dereferencing it
James Almer:
> Since the decoder is not flagged as init cleanup capable, hevc_decode_free()
> is being called manually if the hevc_decode_extradata() call fails at the end
> of hevc_decode_init().
> In a frame threading scenario, however, if AVCodec->init() returns an error,
> ff_frame_thread_free() will be called regardless of the above flag being set
> or not, resulting in hevc_decode_free() being called a second time for the
> same context.
>
> Solve this by ensuring pointers are not dereferenced if they are NULL, and
> setting the decoder as init cleanup capable.
>
> Fixes ticket #9099.
>
> Signed-off-by: James Almer
> ---
> Maybe ff_frame_thread_free() should not call AVCodec->close() for thread
> contexts
> where AVCodec->init() failed and FF_CODEC_CAP_INIT_CLEANUP is not set?
>
Fixing this has been on my to-do list. (The situation is even worse than
you describe it: It is possible that AVCodec->close is called on an
AVCodecContext whose private_data couldn't be allocated.)
> libavcodec/hevcdec.c | 9 -
> 1 file changed, 4 insertions(+), 5 deletions(-)
>
> diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c
> index 92eb888033..898dac8cbb 100644
> --- a/libavcodec/hevcdec.c
> +++ b/libavcodec/hevcdec.c
> @@ -3417,6 +3417,7 @@ static av_cold int hevc_decode_free(AVCodecContext
> *avctx)
> av_freep(>sh.offset);
> av_freep(>sh.size);
>
> +if (s->HEVClcList && s->sList) {
> for (i = 1; i < s->threads_number; i++) {
> HEVCLocalContext *lc = s->HEVClcList[i];
> if (lc) {
> @@ -3424,9 +3425,8 @@ static av_cold int hevc_decode_free(AVCodecContext
> *avctx)
> av_freep(>sList[i]);
> }
> }
> -if (s->HEVClc == s->HEVClcList[0])
> -s->HEVClc = NULL;
> -av_freep(>HEVClcList[0]);
> +}
> +av_freep(>HEVClc);
> av_freep(>HEVClcList);
> av_freep(>sList);
>
> @@ -3622,7 +3622,6 @@ static av_cold int hevc_decode_init(AVCodecContext
> *avctx)
> if (avctx->extradata_size > 0 && avctx->extradata) {
> ret = hevc_decode_extradata(s, avctx->extradata,
> avctx->extradata_size, 1);
> if (ret < 0) {
> -hevc_decode_free(avctx);
> return ret;
> }
> }
> @@ -3673,7 +3672,7 @@ AVCodec ff_hevc_decoder = {
> .capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY |
> AV_CODEC_CAP_SLICE_THREADS |
> AV_CODEC_CAP_FRAME_THREADS,
> .caps_internal = FF_CODEC_CAP_INIT_THREADSAFE |
> FF_CODEC_CAP_EXPORTS_CROPPING |
> - FF_CODEC_CAP_ALLOCATE_PROGRESS,
> + FF_CODEC_CAP_ALLOCATE_PROGRESS |
> FF_CODEC_CAP_INIT_CLEANUP,
> .profiles = NULL_IF_CONFIG_SMALL(ff_hevc_profiles),
> .hw_configs= (const AVCodecHWConfigInternal *const []) {
> #if CONFIG_HEVC_DXVA2_HWACCEL
>
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH 3/8] avformat/mov: simplify size code in probing a bit
On Sat, Feb 06, 2021 at 11:06:47PM +0100, Paul B Mahol wrote:
> LGTM
will apply
thx
{...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
No great genius has ever existed without some touch of madness. -- Aristotle
signature.asc
Description: PGP signature
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] avcodec/hevcdec: check that the local context list exists before dereferencing it
Since the decoder is not flagged as init cleanup capable, hevc_decode_free()
is being called manually if the hevc_decode_extradata() call fails at the end
of hevc_decode_init().
In a frame threading scenario, however, if AVCodec->init() returns an error,
ff_frame_thread_free() will be called regardless of the above flag being set
or not, resulting in hevc_decode_free() being called a second time for the
same context.
Solve this by ensuring pointers are not dereferenced if they are NULL, and
setting the decoder as init cleanup capable.
Fixes ticket #9099.
Signed-off-by: James Almer
---
Maybe ff_frame_thread_free() should not call AVCodec->close() for thread
contexts
where AVCodec->init() failed and FF_CODEC_CAP_INIT_CLEANUP is not set?
libavcodec/hevcdec.c | 9 -
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c
index 92eb888033..898dac8cbb 100644
--- a/libavcodec/hevcdec.c
+++ b/libavcodec/hevcdec.c
@@ -3417,6 +3417,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx)
av_freep(>sh.offset);
av_freep(>sh.size);
+if (s->HEVClcList && s->sList) {
for (i = 1; i < s->threads_number; i++) {
HEVCLocalContext *lc = s->HEVClcList[i];
if (lc) {
@@ -3424,9 +3425,8 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx)
av_freep(>sList[i]);
}
}
-if (s->HEVClc == s->HEVClcList[0])
-s->HEVClc = NULL;
-av_freep(>HEVClcList[0]);
+}
+av_freep(>HEVClc);
av_freep(>HEVClcList);
av_freep(>sList);
@@ -3622,7 +3622,6 @@ static av_cold int hevc_decode_init(AVCodecContext *avctx)
if (avctx->extradata_size > 0 && avctx->extradata) {
ret = hevc_decode_extradata(s, avctx->extradata,
avctx->extradata_size, 1);
if (ret < 0) {
-hevc_decode_free(avctx);
return ret;
}
}
@@ -3673,7 +3672,7 @@ AVCodec ff_hevc_decoder = {
.capabilities = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY |
AV_CODEC_CAP_SLICE_THREADS |
AV_CODEC_CAP_FRAME_THREADS,
.caps_internal = FF_CODEC_CAP_INIT_THREADSAFE |
FF_CODEC_CAP_EXPORTS_CROPPING |
- FF_CODEC_CAP_ALLOCATE_PROGRESS,
+ FF_CODEC_CAP_ALLOCATE_PROGRESS |
FF_CODEC_CAP_INIT_CLEANUP,
.profiles = NULL_IF_CONFIG_SMALL(ff_hevc_profiles),
.hw_configs= (const AVCodecHWConfigInternal *const []) {
#if CONFIG_HEVC_DXVA2_HWACCEL
--
2.30.0
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] libsvtav1: Add lp option
> > It is very non-obvious what "lp" means, "logical_processors" would be > better IMO. > Okay, I will change it to `-logical_processors` ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] libsvtav1: Add logical_processors option
From: Christopher Degawa
Equivalent to the --lp option for SvtAv1EncApp, and is the only way
to control how much cpu power svt-av1 uses for now
Not using thread_count as it would be preferable to reserve that until
svt-av1 properly implements a threads option
0 == getconf _NPROCESSORS_ONLN
Signed-off-by: Christopher Degawa
---
libavcodec/libsvtav1.c | 7 +++
1 file changed, 7 insertions(+)
diff --git a/libavcodec/libsvtav1.c b/libavcodec/libsvtav1.c
index eb6043bcac..2296735f25 100644
--- a/libavcodec/libsvtav1.c
+++ b/libavcodec/libsvtav1.c
@@ -71,6 +71,8 @@ typedef struct SvtContext {
int tile_columns;
int tile_rows;
+
+unsigned logical_processors;
} SvtContext;
static const struct {
@@ -218,6 +220,8 @@ static int config_enc_params(EbSvtAv1EncConfiguration
*param,
param->tile_columns = svt_enc->tile_columns;
param->tile_rows= svt_enc->tile_rows;
+param->logical_processors = svt_enc->logical_processors;
+
return 0;
}
@@ -533,6 +537,9 @@ static const AVOption options[] = {
{ "tile_columns", "Log2 of number of tile columns to use",
OFFSET(tile_columns), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 4, VE},
{ "tile_rows", "Log2 of number of tile rows to use", OFFSET(tile_rows),
AV_OPT_TYPE_INT, {.i64 = 0}, 0, 6, VE},
+{ "logical_processors", "Number of logical processors to run the encoder
on, threads are managed by the OS scheduler", OFFSET(logical_processors),
+ AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, VE },
+
{NULL},
};
--
2.25.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH] lavu/tx: support in-place FFT transforms
This commit adds support for in-place FFT transforms. Since our
internal transforms were all in-place anyway, this only changes
the permutation on the input.
Unfortunately, research papers were of no help here. All focused
on dry hardware implementations, where permutes are free, or on
software implementations where binary bloat is of no concern so
storing dozen times the transforms for each permutation and version
is not considered bad practice.
Still, for a pure C implementation, it's only around 28% slower
than the multi-megabyte FFTW3 in unaligned mode.
Unlike a closed permutation like with PFA, split-radix FFT bit-reversals
contain multiple NOPs, multiple simple swaps, and a few chained swaps,
so regular single-loop single-state permute loops were not possible.
Instead, we filter out parts of the input indices which are redundant.
This allows for a single branch, and with some clever AVX512 asm,
could possibly be SIMD'd without refactoring.
The inplace_idx array is guaranteed to never be larger than the
revtab array, and in practice only requires around log2(len) entries.
The power-of-two MDCTs can be done in-place as well. And it's
possible to eliminate a copy in the compound MDCTs too, however
it'll be slower than doing them out of place, and we'd need to dirty
the input array.
Patch attached.
>From 29e162de51aa71a5bd1d7e85406a053785c08e76 Mon Sep 17 00:00:00 2001
From: Lynne
Date: Wed, 10 Feb 2021 17:58:22 +0100
Subject: [PATCH] lavu/tx: support in-place FFT transforms
This commit adds support for in-place FFT transforms. Since our
internal transforms were all in-place anyway, this only changes
the permutation on the input.
Unfortunately, research papers were of no help here. All focused
on dry hardware implementations, where permutes are free, or on
software implementations where binary bloat is of no concern so
storing dozen times the transforms for each permutation and version
is not considered bad practice.
Still, for a pure C implementation, it's only around 28% slower
than the multi-megabyte FFTW3 in unaligned mode.
Unlike a closed permutation like with PFA, split-radix FFT bit-reversals
contain multiple NOPs, multiple simple swaps, and a few chained swaps,
so regular single-loop single-state permute loops were not possible.
Instead, we filter out parts of the input indices which are redundant.
This allows for a single branch, and with some clever AVX512 asm,
could possibly be SIMD'd without refactoring.
The inplace_idx array is guaranteed to never be larger than the
revtab array, and in practice only requires around log2(len) entries.
The power-of-two MDCTs can be done in-place as well. And it's
possible to eliminate a copy in the compound MDCTs too, however
it'll be slower than doing them out of place, and we'd need to dirty
the input array.
---
libavutil/tx.c | 37 +
libavutil/tx.h | 14 +-
libavutil/tx_priv.h | 9 ++---
libavutil/tx_template.c | 33 ++---
4 files changed, 86 insertions(+), 7 deletions(-)
diff --git a/libavutil/tx.c b/libavutil/tx.c
index 3b0568a5e1..49d5e125ae 100644
--- a/libavutil/tx.c
+++ b/libavutil/tx.c
@@ -107,6 +107,42 @@ int ff_tx_gen_ptwo_revtab(AVTXContext *s)
return 0;
}
+int ff_tx_gen_ptwo_inplace_revtab_idx(AVTXContext *s)
+{
+int nb_inplace_idx = 0;
+
+if (!(s->inplace_idx = av_malloc(s->m*sizeof(*s->inplace_idx
+return AVERROR(ENOMEM);
+
+for (int d = 1; d < s->m; d++) {
+int src = d;
+int dst = s->revtab[src];
+
+if (dst <= src)
+continue;
+
+int found = 0;
+int start_src = src;
+do {
+src = dst;
+for (int j = 0; j < nb_inplace_idx; j++) {
+if (dst == s->inplace_idx[j]) {
+found = 1;
+break;
+}
+}
+dst = s->revtab[dst];
+} while (dst != start_src && !found);
+
+if (!found)
+s->inplace_idx[nb_inplace_idx++] = start_src;
+}
+
+s->inplace_idx[nb_inplace_idx++] = 0;
+
+return 0;
+}
+
av_cold void av_tx_uninit(AVTXContext **ctx)
{
if (!(*ctx))
@@ -115,6 +151,7 @@ av_cold void av_tx_uninit(AVTXContext **ctx)
av_free((*ctx)->pfatab);
av_free((*ctx)->exptab);
av_free((*ctx)->revtab);
+av_free((*ctx)->inplace_idx);
av_free((*ctx)->tmp);
av_freep(ctx);
diff --git a/libavutil/tx.h b/libavutil/tx.h
index f49eb8c4c7..0cf2434847 100644
--- a/libavutil/tx.h
+++ b/libavutil/tx.h
@@ -93,6 +93,18 @@ enum AVTXType {
*/
typedef void (*av_tx_fn)(AVTXContext *s, void *out, void *in, ptrdiff_t stride);
+/**
+ * Flags for av_tx_init()
+ */
+enum AVTXFlags {
+/**
+ * Performs an in-place transformation on the input. The output parameter
+ * to av_tn_fn() will be ignored. May be unsupported or slower for some
+ * transform types.
+ */
+AV_TX_INPLACE
Re: [FFmpeg-devel] [PATCH] x11grab: capture a window instead of the whole screen
On Sat, 6 Feb 2021, Paul B Mahol wrote:
What happens if you use non-existing window?
I added an error message for this case, but the output video is still created
(empty). Is there a value I can return to avoid this?
On Sun, 7 Feb 2021, Andriy Gelman wrote:
Also the commit title should start with avdevice/xcbgrab: or lavd/xcbgrab:
done
If you try with -show_region 1, the border gets drawn in the wrong place.
fixed
I also fixed the misalignement of draw_mouse I didn't notice last time.
+@item window_id
+Grab this window, instead of the whole screen.
+
I think it would be useful to say how to get a window id.
Probably worth adding what happens if the window size changes or it
disappears.
Done. Recording also ends when the window is iconified or reduced in size beyond
the video size.
For now I disabled follow_mouse and select_region when window_id is given.
xcb_window_t is uint32_t, so max should be UINT32_MAX?
Indeed.
@@ -825,6 +827,9 @@ static av_cold int xcbgrab_read_header(AVFormatContext
*s)
return AVERROR(EIO);
}
+if (c->window_id == 0)
+c->window_id = c->screen->root;
+
Can 0 be a valid window_id? Maybe it's better to use -1 for default. I saw the
function xcb_generate_id() returns -1 on error.
0 is not a valid x11 identifier, but it's better to use the XCB_NONE macro for
it (in Xlib and higher-level libraries, it's "None")
Thanks!
Thanks,
--
Andriy
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".From e13c1be7abd6989b3ad80fd8086fe6a0819fd810 Mon Sep 17 00:00:00 2001
From: sgerwk
Date: Wed, 10 Feb 2021 17:36:00 +0100
Subject: [PATCH] libavdevice/xcbgrab: option for grabbing a window instead of
desktop
---
doc/indevs.texi | 14 +++-
libavdevice/xcbgrab.c | 79 ---
2 files changed, 72 insertions(+), 21 deletions(-)
diff --git a/doc/indevs.texi b/doc/indevs.texi
index 3924d03..48fd2b1 100644
--- a/doc/indevs.texi
+++ b/doc/indevs.texi
@@ -1564,8 +1564,20 @@ With @var{follow_mouse}:
ffmpeg -f x11grab -follow_mouse centered -show_region 1 -framerate 25
-video_size cif -i :0.0 out.mpg
@end example
+@item window_id
+Grab this window, instead of the whole screen.
+
+The id of a window can be found by xwininfo(1), possibly with options -tree and
+-root.
+
+If the window is later enlarged, the new area is not recorded. Video ends when
+the window is closed, unmapped (i.e., iconified) or shrunk beyond the video
+size (which defaults to the initial window size).
+
+This option disables options @option{follow_mouse} and @option{select_region}.
+
@item video_size
-Set the video frame size. Default is the full desktop.
+Set the video frame size. Default is the full desktop or window.
@item grab_x
@item grab_y
diff --git a/libavdevice/xcbgrab.c b/libavdevice/xcbgrab.c
index be5d5ea..7697090 100644
--- a/libavdevice/xcbgrab.c
+++ b/libavdevice/xcbgrab.c
@@ -60,6 +60,8 @@ typedef struct XCBGrabContext {
AVRational time_base;
int64_t frame_duration;
+xcb_window_t window_id;
+int win_x, win_y;
int x, y;
int width, height;
int frame_size;
@@ -82,6 +84,7 @@ typedef struct XCBGrabContext {
#define OFFSET(x) offsetof(XCBGrabContext, x)
#define D AV_OPT_FLAG_DECODING_PARAM
static const AVOption options[] = {
+{ "window_id", "Window to capture", OFFSET(window_id), AV_OPT_TYPE_INT, {
.i64 = XCB_NONE }, 0, UINT32_MAX, D },
{ "x", "Initial x coordinate.", OFFSET(x), AV_OPT_TYPE_INT, { .i64 = 0 },
0, INT_MAX, D },
{ "y", "Initial y coordinate.", OFFSET(y), AV_OPT_TYPE_INT, { .i64 = 0 },
0, INT_MAX, D },
{ "grab_x", "Initial x coordinate.", OFFSET(x), AV_OPT_TYPE_INT, { .i64 =
0 }, 0, INT_MAX, D },
@@ -157,7 +160,7 @@ static int xcbgrab_frame(AVFormatContext *s, AVPacket *pkt)
XCBGrabContext *c = s->priv_data;
xcb_get_image_cookie_t iq;
xcb_get_image_reply_t *img;
-xcb_drawable_t drawable = c->screen->root;
+xcb_drawable_t drawable = c->window_id;
xcb_generic_error_t *e = NULL;
uint8_t *data;
int length;
@@ -267,7 +270,7 @@ static int xcbgrab_frame_shm(AVFormatContext *s, AVPacket
*pkt)
XCBGrabContext *c = s->priv_data;
xcb_shm_get_image_cookie_t iq;
xcb_shm_get_image_reply_t *img;
-xcb_drawable_t drawable = c->screen->root;
+xcb_drawable_t drawable = c->window_id;
xcb_generic_error_t *e = NULL;
AVBufferRef *buf;
xcb_shm_seg_t segment;
@@ -355,17 +358,17 @@ static void xcbgrab_draw_mouse(AVFormatContext *s,
AVPacket *pkt,
cx = ci->x - ci->xhot;
cy = ci->y - ci->yhot;
-x = FFMAX(cx, gr->x);
-y = FFMAX(cy, gr->y);
+x = FFMAX(cx, gr->win_x + gr->x);
+y = FFMAX(cy, gr->win_y + gr->y);
-w = FFMIN(cx + ci->width, gr->x + gr->width) - x;
Re: [FFmpeg-devel] [PATCH] [RFC][v3] Tech Resolution Process
Feb 10, 2021, 14:34 by an...@khirnov.net: > Quoting Lynne (2021-02-10 14:15:29) > >> Feb 10, 2021, 11:45 by an...@khirnov.net: >> >> > Quoting Lynne (2021-02-08 20:13:39) >> > >> >> Feb 8, 2021, 15:23 by j...@videolan.org: >> >> > + >> >> > +The decisions from the TC are final, until the matters are reopened >> >> > after >> >> > +no less than one year, by either the GA or the TC auto-seizing. >> >> > >> >> >> >> Is there an OR there? Can the question be raised again after more than >> >> a year by anyone, OR at any time by either the GA or the TC? >> >> >> > >> > I'd say after more than a year through the normal process. So best to >> > drop the part after the comma to avoid confusion. >> > >> >> I think an OR should be added. The TC or GA should be able to say >> "oh, we really screwed it up, now it turns out we absolutely need this" >> without waiting a full year. >> > > It's redundant. GA can already vote on whatever it wants, all the rules > exist only because GA agrees on them. > And the cases of "we fucked up bad, absolutely need to change this" > should be rare enough to not need a special rule. > Oh, sorry, I misread it as "drop the comma". If the part after the comma is removed the document looks good to me. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] [RFC][v3] Tech Resolution Process
Quoting Lynne (2021-02-10 14:15:29) > Feb 10, 2021, 11:45 by an...@khirnov.net: > > > Quoting Lynne (2021-02-08 20:13:39) > > > >> Feb 8, 2021, 15:23 by j...@videolan.org: > >> > + > >> > +The decisions from the TC are final, until the matters are reopened > >> > after > >> > +no less than one year, by either the GA or the TC auto-seizing. > >> > > >> > >> Is there an OR there? Can the question be raised again after more than > >> a year by anyone, OR at any time by either the GA or the TC? > >> > > > > I'd say after more than a year through the normal process. So best to > > drop the part after the comma to avoid confusion. > > > > I think an OR should be added. The TC or GA should be able to say > "oh, we really screwed it up, now it turns out we absolutely need this" > without waiting a full year. It's redundant. GA can already vote on whatever it wants, all the rules exist only because GA agrees on them. And the cases of "we fucked up bad, absolutely need to change this" should be rare enough to not need a special rule. -- Anton Khirnov ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] [RFC][v3] Tech Resolution Process
Feb 10, 2021, 11:45 by an...@khirnov.net: > Quoting Lynne (2021-02-08 20:13:39) > >> Feb 8, 2021, 15:23 by j...@videolan.org: >> > + >> > +The decisions from the TC are final, until the matters are reopened after >> > +no less than one year, by either the GA or the TC auto-seizing. >> > >> >> Is there an OR there? Can the question be raised again after more than >> a year by anyone, OR at any time by either the GA or the TC? >> > > I'd say after more than a year through the normal process. So best to > drop the part after the comma to avoid confusion. > I think an OR should be added. The TC or GA should be able to say "oh, we really screwed it up, now it turns out we absolutely need this" without waiting a full year. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avdevice/avdevice: Deprecate AVDevice Capabilities API
Anton Khirnov: > Quoting Andreas Rheinhardt (2021-02-09 09:04:23) >> Can I get an update on how to proceed with this patch? > > It doesn't seem that anyone actually objected to this patch, so go ahead > and push IMO. > >> >> - Andreas >> >> PS: I could already remove all the av_device_capabilities options >> (except the sentinel) at the next major bump, couldn't I? > > Doxy says should not be used by a user, so I guess yes. > Then I could actually remove these options now. - Andreas ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] [RFC][v3] Tech Resolution Process
Quoting Lynne (2021-02-08 20:13:39) > Feb 8, 2021, 15:23 by j...@videolan.org: > > + > > +The decisions from the TC are final, until the matters are reopened after > > +no less than one year, by either the GA or the TC auto-seizing. > > > > Is there an OR there? Can the question be raised again after more than > a year by anyone, OR at any time by either the GA or the TC? I'd say after more than a year through the normal process. So best to drop the part after the comma to avoid confusion. -- Anton Khirnov ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avformat/movenc: remove unnecessary HDR metadata log messages
lgtm ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avformat/movenc: remove unnecessary HDR metadata log messages
Quoting Jan Ekström (2021-02-10 00:31:51) > These made sense before 3ebf449766d4b69a7774c4990fb24fe71c848a3e > when the creation of these boxes was specifically requested by the > user, but now they have become unnecessary as they are just like > many other boxes: If the input has the information required, they > are written, otherwise they are not. > > They were moved to verbose verbosity level (which happens to be > the last level still relatively usable), and now appear either once > (normal MP4 muxing), or thrice (with the faststart flag set) in > any normal MP4 usage, without giving much useful information. Thus, > remove them in their current form. > --- > > For the record, I am not against having f.ex. the mdcv box writing function > warning in case the side data is there, but it lacks the required information. > > But just always logging this stuff, even if the user didn't go anywhere near > streams with such information does not feel like something useful at this > level of verbosity (esp. with the faststart flag causing it to appear thrice). > > Jan > > --- > libavformat/movenc.c | 2 -- > 1 file changed, 2 deletions(-) av_log(NULL is evil, therefore I like this patch -- Anton Khirnov ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v2] Add support for the new key & value API in libaom.
On Wed, Feb 10, 2021 at 9:16 AM Bohan Li wrote: > > Hi Jan, > > Yes the modified patch looks good to me. > Please let me know if there is anything needed from my end. > > Thank you very much! > > Bohan > Cheers, pulled in as 82aab8a4eec33ee92c92c7679a4d7e6f03b109b4 . Jan ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avdevice/avdevice: Deprecate AVDevice Capabilities API
Quoting Andreas Rheinhardt (2021-02-09 09:04:23) > Can I get an update on how to proceed with this patch? It doesn't seem that anyone actually objected to this patch, so go ahead and push IMO. > > - Andreas > > PS: I could already remove all the av_device_capabilities options > (except the sentinel) at the next major bump, couldn't I? Doxy says should not be used by a user, so I guess yes. -- Anton Khirnov ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] libsvtav1: Add lp option
Quoting Christopher Degawa (2021-02-09 19:58:16) > From: Christopher Degawa > > Equivalent to the --lp option for SvtAv1EncApp, and is the only way > to control how much cpu power svt-av1 uses for now > > Not using thread_count as it would be preferable to reserve that until > svt-av1 properly implements a threads option > > 0 == getconf _NPROCESSORS_ONLN It is very non-obvious what "lp" means, "logical_processors" would be better IMO. -- Anton Khirnov ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avfilter: add exposure video filter
Will apply soon. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH] avfilter: add aexciter audio filter
Will apply soon. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 08/10] libavutil: add side data AVDnnBoundingBox for dnn based detect/classify filters
Signed-off-by: Guo, Yejun
---
doc/APIchanges | 2 ++
libavutil/Makefile | 1 +
libavutil/dnn_bbox.h | 68
libavutil/frame.c| 1 +
libavutil/frame.h| 7 +
libavutil/version.h | 2 +-
6 files changed, 80 insertions(+), 1 deletion(-)
create mode 100644 libavutil/dnn_bbox.h
diff --git a/doc/APIchanges b/doc/APIchanges
index 1332694820..7cbfa9fafa 100644
--- a/doc/APIchanges
+++ b/doc/APIchanges
@@ -14,6 +14,8 @@ libavutil: 2017-10-21
API changes, most recent first:
+2021-02-08 - xx - lavu 56.65.100 - frame.h
+ Add AV_FRAME_DATA_DNN_BBOXES
2021-01-26 - xx - lavu 56.64.100 - common.h
Add FFABSU()
diff --git a/libavutil/Makefile b/libavutil/Makefile
index 27bafe9e12..b99cb7084f 100644
--- a/libavutil/Makefile
+++ b/libavutil/Makefile
@@ -23,6 +23,7 @@ HEADERS = adler32.h
\
des.h \
dict.h\
display.h \
+ dnn_bbox.h\
dovi_meta.h \
downmix_info.h\
encryption_info.h \
diff --git a/libavutil/dnn_bbox.h b/libavutil/dnn_bbox.h
new file mode 100644
index 00..50899c4486
--- /dev/null
+++ b/libavutil/dnn_bbox.h
@@ -0,0 +1,68 @@
+/*
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef AVUTIL_DNN_BBOX_H
+#define AVUTIL_DNN_BBOX_H
+
+#include "rational.h"
+
+typedef struct AVDnnBoundingBox {
+/**
+ * Must be set to the size of this data structure (that is,
+ * sizeof(AVDnnBoundingBox)).
+ */
+uint32_t self_size;
+
+/**
+ * Object detection is usually applied to a smaller image that
+ * is scaled down from the original frame.
+ * width and height are attributes of the scaled image, in pixel.
+ */
+int model_input_width;
+int model_input_height;
+
+/**
+ * Distance in pixels from the top edge of the scaled image to top
+ * and bottom, and from the left edge of the scaled image to left and
+ * right, defining the bounding box.
+ */
+int top;
+int left;
+int bottom;
+int right;
+
+/**
+ * Detect result
+ */
+int detect_label;
+AVRational detect_conf;
+
+/**
+ * At most 4 classifications based on the detected bounding box.
+ * For example, we can get max 4 different attributes with 4 different
+ * DNN models on one bounding box.
+ * classify_count is zero if no classification.
+ */
+#define AV_NUM_BBOX_CLASSIFY 4
+uint32_t classify_count;
+int classify_labels[AV_NUM_BBOX_CLASSIFY];
+AVRational classify_confs[AV_NUM_BBOX_CLASSIFY];
+} AVDnnBoundingBox;
+
+#endif
diff --git a/libavutil/frame.c b/libavutil/frame.c
index eab51b6a32..4308507827 100644
--- a/libavutil/frame.c
+++ b/libavutil/frame.c
@@ -852,6 +852,7 @@ const char *av_frame_side_data_name(enum
AVFrameSideDataType type)
case AV_FRAME_DATA_VIDEO_ENC_PARAMS:return "Video encoding
parameters";
case AV_FRAME_DATA_SEI_UNREGISTERED:return "H.26[45] User Data
Unregistered SEI message";
case AV_FRAME_DATA_FILM_GRAIN_PARAMS: return "Film grain
parameters";
+case AV_FRAME_DATA_DNN_BBOXES: return "DNN bounding
boxes";
}
return NULL;
}
diff --git a/libavutil/frame.h b/libavutil/frame.h
index 1aeafef6de..a4dcfd27c9 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -198,6 +198,13 @@ enum AVFrameSideDataType {
* Must be present for every frame which should have film grain applied.
*/
AV_FRAME_DATA_FILM_GRAIN_PARAMS,
+
+/**
+ * Bounding box generated by dnn based filters for object detection and
classification,
+ * the data is an array of AVDnnBoudingBox, the number of array element is
implied by
+ * AVFrameSideData.size / AVDnnBoudingBox.self_size.
+ */
+AV_FRAME_DATA_DNN_BBOXES,
};
enum
[FFmpeg-devel] [PATCH V2 07/10] dnn_backend_openvino.c: allow out_frame as NULL for analytic case
From: Ting Fu
---
libavfilter/dnn/dnn_backend_openvino.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index cca155a52c..5be053b7f8 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -616,7 +616,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel
*model, const char *input_n
return DNN_ERROR;
}
-if (!out_frame) {
+if (!out_frame && model->func_type == DFT_PROCESS_FRAME) {
av_log(ctx, AV_LOG_ERROR, "out frame is NULL when execute model.\n");
return DNN_ERROR;
}
@@ -669,7 +669,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel
*model, const char *i
return DNN_ERROR;
}
-if (!out_frame) {
+if (!out_frame && model->func_type == DFT_PROCESS_FRAME) {
av_log(ctx, AV_LOG_ERROR, "out frame is NULL when async execute
model.\n");
return DNN_ERROR;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 10/10] libavfilter: add filter dnn_detect for object detection
Below are the example steps to do object detection:
1. download and install l_openvino_toolkit_p_2021.1.110.tgz from
https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html
or, We can get source code (tag 2021.1), build and install.
2. export LD_LIBRARY_PATH with openvino settings, for example:
.../deployment_tools/inference_engine/lib/intel64/:.../deployment_tools/inference_engine/external/tbb/lib/
3. rebuild ffmpeg from source code with configure option:
--enable-libopenvino
--extra-cflags='-I.../deployment_tools/inference_engine/include/'
--extra-ldflags='-L.../deployment_tools/inference_engine/lib/intel64'
4. download model files and test image
wget
https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.bin
wget
https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.xml
wget https://github.com/guoyejun/ffmpeg_dnn/raw/main/images/cici.jpg
5. run ffmpeg with:
./ffmpeg -i cici.jpg -vf
dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:input=data:output=detection_out:conf=0.6,showinfo
-f null -
We'll see the detect result as below:
[Parsed_showinfo_1 @ 0x55db3ffb60c0] side data - Dnn bounding boxes:
[Parsed_showinfo_1 @ 0x55db3ffb60c0] index: 0, region: (330/672, 203/384) ->
(356/672, 226/384), label: 1, confidence: 1/1.
[Parsed_showinfo_1 @ 0x55db3ffb60c0] index: 1, region: (291/672, 209/384) ->
(317/672, 231/384), label: 1, confidence: 6917/1.
There are two faces detected with confidence 100% and 69.17%, and
the input image size of the model is 672x384. The two bounding boxes
in this image are (330, 203)->(356, 226) and (291, 209)->(317, 231).
Since the orignal input image size is 2048x1536, so the two bounding
boxese in the original image are
(330/672*2048=1006, 203/384*1536=812) -> (1085, 904) and
(887, 836) -> (966, 924), and we can check them manually.
Signed-off-by: Guo, Yejun
Next, we'll add tensorflow backend and update filter vf_drawbox etc
to visualize the detect result.
---
configure | 1 +
doc/filters.texi | 33 +++
libavfilter/Makefile | 1 +
libavfilter/allfilters.c | 1 +
libavfilter/dnn/dnn_backend_openvino.c | 12 +
libavfilter/dnn_filter_common.c| 7 +
libavfilter/dnn_filter_common.h| 1 +
libavfilter/dnn_interface.h| 6 +-
libavfilter/vf_dnn_detect.c| 356 +
9 files changed, 416 insertions(+), 2 deletions(-)
create mode 100644 libavfilter/vf_dnn_detect.c
diff --git a/configure b/configure
index a76c2ec4ae..2d2668571d 100755
--- a/configure
+++ b/configure
@@ -3548,6 +3548,7 @@ derain_filter_select="dnn"
deshake_filter_select="pixelutils"
deshake_opencl_filter_deps="opencl"
dilation_opencl_filter_deps="opencl"
+dnn_detect_filter_select="dnn"
dnn_processing_filter_select="dnn"
drawtext_filter_deps="libfreetype"
drawtext_filter_suggest="libfontconfig libfribidi"
diff --git a/doc/filters.texi b/doc/filters.texi
index 079bba9a1e..340402e650 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -10073,6 +10073,39 @@ ffmpeg -i INPUT -f lavfi -i
nullsrc=hd720,geq='r=128+80*(sin(sqrt((X-W/2)*(X-W/2
@end example
@end itemize
+@section dnn_detect
+
+Do object detection with deep neural networks.
+
+The filter accepts the following options:
+
+@table @option
+@item dnn_backend
+Specify which DNN backend to use for model loading and execution. This option
accepts
+only openvino now, tensorflow backends will be added.
+
+@item model
+Set path to model file specifying network architecture and its parameters.
+Note that different backends use different file formats.
+
+@item input
+Set the input name of the dnn network.
+
+@item output
+Set the output name of the dnn network.
+
+@item conf
+Set the confidence threshold (default: 0.5).
+
+@item backend_configs
+Set the configs to be passed into backend
+
+@item async
+use DNN async execution if set (default: set),
+roll back to sync execution if the backend does not support async.
+
+@end table
+
@anchor{dnn_processing}
@section dnn_processing
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index b43933be64..6c39e7111b 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -244,6 +244,7 @@ OBJS-$(CONFIG_DILATION_FILTER) +=
vf_neighbor.o
OBJS-$(CONFIG_DILATION_OPENCL_FILTER)+= vf_neighbor_opencl.o opencl.o \
opencl/neighbor.o
OBJS-$(CONFIG_DISPLACE_FILTER) += vf_displace.o framesync.o
+OBJS-$(CONFIG_DNN_DETECT_FILTER) += vf_dnn_detect.o
OBJS-$(CONFIG_DNN_PROCESSING_FILTER) += vf_dnn_processing.o
OBJS-$(CONFIG_DOUBLEWEAVE_FILTER)+= vf_weave.o
OBJS-$(CONFIG_DRAWBOX_FILTER)+= vf_drawbox.o
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
index
[FFmpeg-devel] [PATCH V2 09/10] libavfilter: show side data of dnn bounding box
Signed-off-by: Guo, Yejun
---
libavfilter/f_sidedata.c | 2 ++
libavfilter/vf_showinfo.c | 34 ++
2 files changed, 36 insertions(+)
diff --git a/libavfilter/f_sidedata.c b/libavfilter/f_sidedata.c
index 3757723375..08ee96c4ef 100644
--- a/libavfilter/f_sidedata.c
+++ b/libavfilter/f_sidedata.c
@@ -71,6 +71,7 @@ static const AVOption filt_name##_options[] = { \
{ "S12M_TIMECOD", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_S12M_TIMECODE }, 0, 0, FLAGS, "type" }, \
{ "DYNAMIC_HDR_PLUS", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_DYNAMIC_HDR_PLUS }, 0, 0, FLAGS, "type" }, \
{ "REGIONS_OF_INTEREST","", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_REGIONS_OF_INTEREST}, 0, 0, FLAGS, "type" }, \
+{ "DNN_BOUNDING_BOXES", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_DNN_BBOXES }, 0, 0, FLAGS, "type" }, \
{ "SEI_UNREGISTERED", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_SEI_UNREGISTERED }, 0, 0, FLAGS, "type" }, \
{ NULL } \
}
@@ -100,6 +101,7 @@ static const AVOption filt_name##_options[] = { \
{ "S12M_TIMECOD", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_S12M_TIMECODE }, 0, 0, FLAGS, "type" }, \
{ "DYNAMIC_HDR_PLUS", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_DYNAMIC_HDR_PLUS }, 0, 0, FLAGS, "type" }, \
{ "REGIONS_OF_INTEREST","", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_REGIONS_OF_INTEREST}, 0, 0, FLAGS, "type" }, \
+{ "DNN_BOUNDING_BOXES", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_DNN_BBOXES }, 0, 0, FLAGS, "type" }, \
{ "SEI_UNREGISTERED", "", 0, AV_OPT_TYPE_CONST,
{.i64 = AV_FRAME_DATA_SEI_UNREGISTERED }, 0, 0, FLAGS, "type" }, \
{ NULL } \
}
diff --git a/libavfilter/vf_showinfo.c b/libavfilter/vf_showinfo.c
index 6208892005..67653b17ec 100644
--- a/libavfilter/vf_showinfo.c
+++ b/libavfilter/vf_showinfo.c
@@ -38,6 +38,7 @@
#include "libavutil/timecode.h"
#include "libavutil/mastering_display_metadata.h"
#include "libavutil/video_enc_params.h"
+#include "libavutil/dnn_bbox.h"
#include "avfilter.h"
#include "internal.h"
@@ -153,6 +154,36 @@ static void dump_roi(AVFilterContext *ctx, const
AVFrameSideData *sd)
}
}
+static void dump_dnnbbox(AVFilterContext *ctx, const AVFrameSideData *sd)
+{
+int nb_bbox;
+const AVDnnBoundingBox *bbox;
+uint32_t bbox_size;
+
+bbox = (const AVDnnBoundingBox *)sd->data;
+bbox_size = bbox->self_size;
+if (!bbox_size || sd->size % bbox_size != 0 || bbox_size != sizeof(*bbox))
{
+av_log(ctx, AV_LOG_ERROR, "Invalid AVDnnBoundingBox.self_size.\n");
+return;
+}
+nb_bbox = sd->size / bbox_size;
+
+av_log(ctx, AV_LOG_INFO, "Dnn bounding boxes:\n");
+for (int i = 0; i < nb_bbox; i++) {
+bbox = (const AVDnnBoundingBox *)(sd->data + bbox_size * i);
+av_log(ctx, AV_LOG_INFO, "index: %d, region: (%d/%d, %d/%d) -> (%d/%d,
%d/%d), label: %d, confidence: %d/%d.\n",
+ i, bbox->left, bbox->model_input_width, bbox->top,
bbox->model_input_height,
+ bbox->right, bbox->model_input_width, bbox->bottom,
bbox->model_input_height,
+ bbox->detect_label, bbox->detect_conf.num,
bbox->detect_conf.den);
+if (bbox->classify_count > 0) {
+for (int j = 0; j < bbox->classify_count; j++) {
+av_log(ctx, AV_LOG_INFO, "\t\tclassify: label: %d,
confidence: %d/%d.\n",
+ bbox->classify_labels[j], bbox->classify_confs[j].num,
bbox->classify_confs[j].den);
+}
+}
+}
+}
+
static void dump_mastering_display(AVFilterContext *ctx, const AVFrameSideData
*sd)
{
const AVMasteringDisplayMetadata *mastering_display;
@@ -494,6 +525,9 @@ static int filter_frame(AVFilterLink *inlink, AVFrame
*frame)
case AV_FRAME_DATA_REGIONS_OF_INTEREST:
dump_roi(ctx, sd);
break;
+case AV_FRAME_DATA_DNN_BBOXES:
+dump_dnnbbox(ctx, sd);
+break;
case AV_FRAME_DATA_MASTERING_DISPLAY_METADATA:
dump_mastering_display(ctx, sd);
break;
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 06/10] dnn: add color conversion for analytic case
Signed-off-by: Guo, Yejun
---
libavfilter/dnn/dnn_backend_native.c | 2 +-
libavfilter/dnn/dnn_backend_openvino.c | 23 ++-
libavfilter/dnn/dnn_backend_tf.c | 2 +-
libavfilter/dnn/dnn_io_proc.c | 56 +-
libavfilter/dnn/dnn_io_proc.h | 2 +-
5 files changed, 80 insertions(+), 5 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index be6451367a..3bc253c1ad 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -321,7 +321,7 @@ static DNNReturnType execute_model_native(const DNNModel
*model, const char *inp
if (native_model->model->pre_proc != NULL) {
native_model->model->pre_proc(in_frame, ,
native_model->model->filter_ctx);
} else {
-ff_proc_from_frame_to_dnn(in_frame, , ctx);
+ff_proc_from_frame_to_dnn(in_frame, ,
native_model->model->func_type, ctx);
}
}
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 7c1abb3eeb..cca155a52c 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -99,6 +99,8 @@ static DNNDataType precision_to_datatype(precision_e
precision)
{
case FP32:
return DNN_FLOAT;
+case U8:
+return DNN_UINT8;
default:
av_assert0(!"not supported yet.");
return DNN_FLOAT;
@@ -111,6 +113,8 @@ static int get_datatype_size(DNNDataType dt)
{
case DNN_FLOAT:
return sizeof(float);
+case DNN_UINT8:
+return sizeof(uint8_t);
default:
av_assert0(!"not supported yet.");
return 1;
@@ -152,6 +156,9 @@ static DNNReturnType fill_model_input_ov(OVModel *ov_model,
RequestItem *request
input.channels = dims.dims[1];
input.data = blob_buffer.buffer;
input.dt = precision_to_datatype(precision);
+// all models in openvino open model zoo use BGR as input,
+// change to be an option when necessary.
+input.order = DCO_BGR;
av_assert0(request->task_count <= dims.dims[0]);
for (int i = 0; i < request->task_count; ++i) {
@@ -160,7 +167,7 @@ static DNNReturnType fill_model_input_ov(OVModel *ov_model,
RequestItem *request
if (ov_model->model->pre_proc != NULL) {
ov_model->model->pre_proc(task->in_frame, ,
ov_model->model->filter_ctx);
} else {
-ff_proc_from_frame_to_dnn(task->in_frame, , ctx);
+ff_proc_from_frame_to_dnn(task->in_frame, ,
ov_model->model->func_type, ctx);
}
}
input.data = (uint8_t *)input.data
@@ -290,6 +297,20 @@ static DNNReturnType init_model_ov(OVModel *ov_model,
const char *input_name, co
goto err;
}
+// all models in openvino open model zoo use BGR with range [0.0f, 255.0f]
as input,
+// we don't have a AVPixelFormat to descibe it, so we'll use
AV_PIX_FMT_BGR24 and
+// ask openvino to do the conversion internally.
+// the current supported SR model (frame processing) is generated from
tensorflow model,
+// and its input is Y channel as float with range [0.0f, 1.0f], so do not
set for this case.
+// TODO: we need to get a final clear solution with all
backends/formats considered.
+if (ov_model->model->func_type != DFT_PROCESS_FRAME) {
+status = ie_network_set_input_precision(ov_model->network, input_name,
U8);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to set input precision as U8 for
%s\n", input_name);
+return DNN_ERROR;
+}
+}
+
status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n");
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index e7e5f221f3..750a476726 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -744,7 +744,7 @@ static DNNReturnType execute_model_tf(const DNNModel
*model, const char *input_n
if (tf_model->model->pre_proc != NULL) {
tf_model->model->pre_proc(in_frame, ,
tf_model->model->filter_ctx);
} else {
-ff_proc_from_frame_to_dnn(in_frame, , ctx);
+ff_proc_from_frame_to_dnn(in_frame, ,
tf_model->model->func_type, ctx);
}
}
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index bee1423342..e104cc5064 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -21,6 +21,7 @@
#include "dnn_io_proc.h"
#include "libavutil/imgutils.h"
#include "libswscale/swscale.h"
+#include "libavutil/avassert.h"
DNNReturnType ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void
*log_ctx)
{
@@ -92,7 +93,7 @@ DNNReturnType
[FFmpeg-devel] [PATCH V2 05/10] dnn_interface.h: add enum DNNColorOrder
the data type and order together decide the color format, we could
not use AVPixelFormat directly because not all the possible formats
are covered by it.
Signed-off-by: Guo, Yejun
---
libavfilter/dnn_interface.h | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h
index 2fb9b15676..d3a0c58a61 100644
--- a/libavfilter/dnn_interface.h
+++ b/libavfilter/dnn_interface.h
@@ -36,6 +36,11 @@ typedef enum {DNN_NATIVE, DNN_TF, DNN_OV} DNNBackendType;
typedef enum {DNN_FLOAT = 1, DNN_UINT8 = 4} DNNDataType;
+typedef enum {
+DCO_NONE,
+DCO_BGR,
+} DNNColorOrder;
+
typedef enum {
DAST_FAIL, // something wrong
DAST_EMPTY_QUEUE, // no more inference result to get
@@ -52,8 +57,10 @@ typedef enum {
typedef struct DNNData{
void *data;
-DNNDataType dt;
int width, height, channels;
+// dt and order together decide the color format
+DNNDataType dt;
+DNNColorOrder order;
} DNNData;
typedef struct DNNModel{
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 04/10] dnn: add function type for model
So the backend knows the usage of model is for frame processing,
detect, classify, etc. Each function type has different behavior
in backend when handling the input/output data of the model.
Signed-off-by: Guo, Yejun
---
libavfilter/dnn/dnn_backend_native.c | 3 ++-
libavfilter/dnn/dnn_backend_native.h | 2 +-
libavfilter/dnn/dnn_backend_openvino.c | 3 ++-
libavfilter/dnn/dnn_backend_openvino.h | 2 +-
libavfilter/dnn/dnn_backend_tf.c | 5 +++--
libavfilter/dnn/dnn_backend_tf.h | 2 +-
libavfilter/dnn_filter_common.c| 4 ++--
libavfilter/dnn_filter_common.h| 2 +-
libavfilter/dnn_interface.h| 11 ++-
libavfilter/vf_derain.c| 2 +-
libavfilter/vf_dnn_processing.c| 2 +-
libavfilter/vf_sr.c| 2 +-
12 files changed, 26 insertions(+), 14 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_native.c
b/libavfilter/dnn/dnn_backend_native.c
index 87f3568cc2..be6451367a 100644
--- a/libavfilter/dnn/dnn_backend_native.c
+++ b/libavfilter/dnn/dnn_backend_native.c
@@ -112,7 +112,7 @@ static DNNReturnType get_output_native(void *model, const
char *input_name, int
// layers_num,layer_type,layer_parameterss,layer_type,layer_parameters...
// For CONV layer: activation_function, input_num, output_num, kernel_size,
kernel, biases
// For DEPTH_TO_SPACE layer: block_size
-DNNModel *ff_dnn_load_model_native(const char *model_filename, const char
*options, AVFilterContext *filter_ctx)
+DNNModel *ff_dnn_load_model_native(const char *model_filename, DNNFunctionType
func_type, const char *options, AVFilterContext *filter_ctx)
{
DNNModel *model = NULL;
char header_expected[] = "FFMPEGDNNNATIVE";
@@ -256,6 +256,7 @@ DNNModel *ff_dnn_load_model_native(const char
*model_filename, const char *optio
model->get_input = _input_native;
model->get_output = _output_native;
model->filter_ctx = filter_ctx;
+model->func_type = func_type;
return model;
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 5c8ce82b35..d313c48f3a 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -128,7 +128,7 @@ typedef struct NativeModel{
int32_t operands_num;
} NativeModel;
-DNNModel *ff_dnn_load_model_native(const char *model_filename, const char
*options, AVFilterContext *filter_ctx);
+DNNModel *ff_dnn_load_model_native(const char *model_filename, DNNFunctionType
func_type, const char *options, AVFilterContext *filter_ctx);
DNNReturnType ff_dnn_execute_model_native(const DNNModel *model, const char
*input_name, AVFrame *in_frame,
const char **output_names, uint32_t
nb_output, AVFrame *out_frame);
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index ed41b721fc..7c1abb3eeb 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -524,7 +524,7 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
return ret;
}
-DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char
*options, AVFilterContext *filter_ctx)
+DNNModel *ff_dnn_load_model_ov(const char *model_filename, DNNFunctionType
func_type, const char *options, AVFilterContext *filter_ctx)
{
DNNModel *model = NULL;
OVModel *ov_model = NULL;
@@ -572,6 +572,7 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename,
const char *options,
model->get_output = _output_ov;
model->options = options;
model->filter_ctx = filter_ctx;
+model->func_type = func_type;
return model;
diff --git a/libavfilter/dnn/dnn_backend_openvino.h
b/libavfilter/dnn/dnn_backend_openvino.h
index 23b819440e..a484a7be32 100644
--- a/libavfilter/dnn/dnn_backend_openvino.h
+++ b/libavfilter/dnn/dnn_backend_openvino.h
@@ -29,7 +29,7 @@
#include "../dnn_interface.h"
-DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char
*options, AVFilterContext *filter_ctx);
+DNNModel *ff_dnn_load_model_ov(const char *model_filename, DNNFunctionType
func_type, const char *options, AVFilterContext *filter_ctx);
DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, const char
*input_name, AVFrame *in_frame,
const char **output_names, uint32_t
nb_output, AVFrame *out_frame);
diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c
index 71a2a308b5..e7e5f221f3 100644
--- a/libavfilter/dnn/dnn_backend_tf.c
+++ b/libavfilter/dnn/dnn_backend_tf.c
@@ -580,7 +580,7 @@ static DNNReturnType load_native_model(TFModel *tf_model,
const char *model_file
DNNModel *model = NULL;
NativeModel *native_model;
-model = ff_dnn_load_model_native(model_filename, NULL, NULL);
+model = ff_dnn_load_model_native(model_filename, DFT_PROCESS_FRAME, NULL,
NULL);
if (!model){
[FFmpeg-devel] [PATCH V2 03/10] dnn: extract common functions used by different filters
Signed-off-by: Guo, Yejun
---
libavfilter/Makefile| 1 +
libavfilter/dnn_filter_common.c | 106
libavfilter/dnn_filter_common.h | 59 ++
libavfilter/vf_derain.c | 46 +++---
libavfilter/vf_dnn_processing.c | 93 +---
libavfilter/vf_sr.c | 53
6 files changed, 201 insertions(+), 157 deletions(-)
create mode 100644 libavfilter/dnn_filter_common.c
create mode 100644 libavfilter/dnn_filter_common.h
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index 3ec28df411..b43933be64 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -27,6 +27,7 @@ OBJS-$(HAVE_THREADS) += pthread.o
# subsystems
OBJS-$(CONFIG_QSVVPP)+= qsvvpp.o
OBJS-$(CONFIG_SCENE_SAD) += scene_sad.o
+OBJS-$(CONFIG_DNN) += dnn_filter_common.o
include $(SRC_PATH)/libavfilter/dnn/Makefile
# audio filters
diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c
new file mode 100644
index 00..5d0d7d3b90
--- /dev/null
+++ b/libavfilter/dnn_filter_common.c
@@ -0,0 +1,106 @@
+/*
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include "dnn_filter_common.h"
+
+int ff_dnn_init(DnnContext *ctx, AVFilterContext *filter_ctx)
+{
+if (!ctx->model_filename) {
+av_log(filter_ctx, AV_LOG_ERROR, "model file for network is not
specified\n");
+return AVERROR(EINVAL);
+}
+if (!ctx->model_inputname) {
+av_log(filter_ctx, AV_LOG_ERROR, "input name of the model network is
not specified\n");
+return AVERROR(EINVAL);
+}
+if (!ctx->model_outputname) {
+av_log(filter_ctx, AV_LOG_ERROR, "output name of the model network is
not specified\n");
+return AVERROR(EINVAL);
+}
+
+ctx->dnn_module = ff_get_dnn_module(ctx->backend_type);
+if (!ctx->dnn_module) {
+av_log(filter_ctx, AV_LOG_ERROR, "could not create DNN module for
requested backend\n");
+return AVERROR(ENOMEM);
+}
+if (!ctx->dnn_module->load_model) {
+av_log(filter_ctx, AV_LOG_ERROR, "load_model for network is not
specified\n");
+return AVERROR(EINVAL);
+}
+
+ctx->model = (ctx->dnn_module->load_model)(ctx->model_filename,
ctx->backend_options, filter_ctx);
+if (!ctx->model) {
+av_log(filter_ctx, AV_LOG_ERROR, "could not load DNN model\n");
+return AVERROR(EINVAL);
+}
+
+if (!ctx->dnn_module->execute_model_async && ctx->async) {
+ctx->async = 0;
+av_log(filter_ctx, AV_LOG_WARNING, "this backend does not support
async execution, roll back to sync.\n");
+}
+
+#if !HAVE_PTHREAD_CANCEL
+if (ctx->async) {
+ctx->async = 0;
+av_log(filter_ctx, AV_LOG_WARNING, "pthread is not supported, roll
back to sync.\n");
+}
+#endif
+
+return 0;
+}
+
+DNNReturnType ff_dnn_get_input(DnnContext *ctx, DNNData *input)
+{
+return ctx->model->get_input(ctx->model->model, input,
ctx->model_inputname);
+}
+
+DNNReturnType ff_dnn_get_output(DnnContext *ctx, int input_width, int
input_height, int *output_width, int *output_height)
+{
+return ctx->model->get_output(ctx->model->model, ctx->model_inputname,
input_width, input_height,
+ctx->model_outputname, output_width,
output_height);
+}
+
+DNNReturnType ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame
*out_frame)
+{
+return (ctx->dnn_module->execute_model)(ctx->model, ctx->model_inputname,
in_frame,
+(const char
**)>model_outputname, 1, out_frame);
+}
+
+DNNReturnType ff_dnn_execute_model_async(DnnContext *ctx, AVFrame *in_frame,
AVFrame *out_frame)
+{
+return (ctx->dnn_module->execute_model_async)(ctx->model,
ctx->model_inputname, in_frame,
+ (const char
**)>model_outputname, 1, out_frame);
+}
+
+DNNAsyncStatusType ff_dnn_get_async_result(DnnContext *ctx, AVFrame
**in_frame, AVFrame **out_frame)
+{
+return (ctx->dnn_module->get_async_result)(ctx->model, in_frame,
out_frame);
+}
+
+DNNReturnType ff_dnn_flush(DnnContext
[FFmpeg-devel] [PATCH V2 01/10] dnn_backend_openvino.c: fix mismatch between ffmpeg(NHWC) and openvino(NCHW)
Signed-off-by: Guo, Yejun
---
libavfilter/dnn/dnn_backend_openvino.c | 21 +
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index beca256390..48f5ba50be 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -250,7 +250,7 @@ static void infer_completion_callback(void *args)
}
}
-static DNNReturnType init_model_ov(OVModel *ov_model)
+static DNNReturnType init_model_ov(OVModel *ov_model, const char *input_name,
const char *output_name)
{
OVContext *ctx = _model->ctx;
IEStatusCode status;
@@ -276,6 +276,19 @@ static DNNReturnType init_model_ov(OVModel *ov_model)
goto err;
}
+// The order of dims in the openvino is fixed and it is always NCHW for
4-D data.
+// while we pass NHWC data from FFmpeg to openvino
+status = ie_network_set_input_layout(ov_model->network, input_name, NHWC);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for input
%s\n", input_name);
+goto err;
+}
+status = ie_network_set_output_layout(ov_model->network, output_name,
NHWC);
+if (status != OK) {
+av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for output
%s\n", output_name);
+goto err;
+}
+
status = ie_core_load_network(ov_model->core, ov_model->network,
ctx->options.device_type, , _model->exe_network);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n");
@@ -482,7 +495,7 @@ static DNNReturnType get_output_ov(void *model, const char
*input_name, int inpu
}
if (!ov_model->exe_network) {
-if (init_model_ov(ov_model) != DNN_SUCCESS) {
+if (init_model_ov(ov_model, input_name, output_name) != DNN_SUCCESS) {
av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
return DNN_ERROR;
}
@@ -598,7 +611,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel
*model, const char *input_n
}
if (!ov_model->exe_network) {
-if (init_model_ov(ov_model) != DNN_SUCCESS) {
+if (init_model_ov(ov_model, input_name, output_names[0]) !=
DNN_SUCCESS) {
av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
return DNN_ERROR;
}
@@ -645,7 +658,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel
*model, const char *i
}
if (!ov_model->exe_network) {
-if (init_model_ov(ov_model) != DNN_SUCCESS) {
+if (init_model_ov(ov_model, input_name, output_names[0]) !=
DNN_SUCCESS) {
av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network
or inference request\n");
return DNN_ERROR;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [PATCH V2 02/10] dnn_backend_openvino.c: fix multi-thread issue for async execution
once we mark done for the task in function infer_completion_callback,
the task is possible to be release in function ff_dnn_get_async_result_ov
in another thread just after it, so we need to record request queue
first, instead of using task->ov_model->request_queue later.
Signed-off-by: Guo, Yejun
---
libavfilter/dnn/dnn_backend_openvino.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 48f5ba50be..ed41b721fc 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -178,6 +178,7 @@ static void infer_completion_callback(void *args)
IEStatusCode status;
RequestItem *request = args;
TaskItem *task = request->tasks[0];
+SafeQueue *requestq = task->ov_model->request_queue;
ie_blob_t *output_blob = NULL;
ie_blob_buffer_t blob_buffer;
DNNData output;
@@ -243,7 +244,7 @@ static void infer_completion_callback(void *args)
request->task_count = 0;
if (task->async) {
-if (ff_safe_queue_push_back(task->ov_model->request_queue, request) <
0) {
+if (ff_safe_queue_push_back(requestq, request) < 0) {
av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n");
return;
}
--
2.17.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH 08/10] libavutil: add side data AVDnnBoundingBox for dnn based detect/classify filters
> -Original Message- > From: Guo, Yejun > Sent: 2021年2月10日 12:59 > To: ffmpeg-devel@ffmpeg.org > Cc: Guo, Yejun > Subject: [PATCH 08/10] libavutil: add side data AVDnnBoundingBox for dnn > based detect/classify filters > > Signed-off-by: Guo, Yejun > --- > doc/APIchanges | 2 ++ > libavutil/Makefile | 1 + > libavutil/dnn_bbox.h | 68 > > libavutil/frame.c| 1 + > libavutil/frame.h| 7 + > libavutil/version.h | 2 +- > 6 files changed, 80 insertions(+), 1 deletion(-) create mode 100644 > libavutil/dnn_bbox.h > > diff --git a/doc/APIchanges b/doc/APIchanges index 1332694820..7cbfa9fafa > 100644 > --- a/doc/APIchanges > +++ b/doc/APIchanges > @@ -14,6 +14,8 @@ libavutil: 2017-10-21 > > > API changes, most recent first: > +2021-02-08 - xx - lavu 56.65.100 - frame.h > + Add AV_FRAME_DATA_DNN_BBOXES > > 2021-01-26 - xx - lavu 56.64.100 - common.h >Add FFABSU() > diff --git a/libavutil/Makefile b/libavutil/Makefile index > 27bafe9e12..b99cb7084f 100644 > --- a/libavutil/Makefile > +++ b/libavutil/Makefile > @@ -23,6 +23,7 @@ HEADERS = adler32.h > \ >des.h > \ >dict.h > \ >display.h > \ > + dnn_bbox.h > \ >dovi_meta.h > \ >downmix_info.h > \ >encryption_info.h > \ > diff --git a/libavutil/dnn_bbox.h b/libavutil/dnn_bbox.h new file mode 100644 > index 00..ae68f9e2e2 > --- /dev/null > +++ b/libavutil/dnn_bbox.h > @@ -0,0 +1,68 @@ > +/* > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA > +02110-1301 USA */ > + > +#ifndef AVFILTER_DNN_BBOX_H > +#define AVFILTER_DNN_BBOX_H the macro name is not correct and caught by patchwork, will fix it and send the whole patch set as V2, thanks. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
