Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

Pushed.

Thanks.

2018-06-07 12:29 GMT-03:00 Sergey Lavrushkin :
> 2018-06-06 17:22 GMT+03:00 Pedro Arthur :
>>
>> Hi,
>>
>> 2018-06-05 20:23 GMT-03:00 Sergey Lavrushkin :
>> > Here is the patch, that fixes described issues.
>> When I try to run (video input), when tf is not enabled in configure it
>> crashes.
>>
>>
>> $ffmpeg -i in.mp4 -vf srcnn=dnn_backend=tensorflow out.mp4
>>
>> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
>> developers
>>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>>   configuration:
>>   libavutil  56. 18.102 / 56. 18.102
>>   libavcodec 58. 19.105 / 58. 19.105
>>   libavformat58. 17.100 / 58. 17.100
>>   libavdevice58.  4.100 / 58.  4.100
>>   libavfilter 7. 25.100 /  7. 25.100
>>   libswscale  5.  2.100 /  5.  2.100
>>   libswresample   3.  2.100 /  3.  2.100
>> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mp4':
>>   Metadata:
>> major_brand : isom
>> minor_version   : 512
>> compatible_brands: isomiso2mp41
>> encoder : Lavf58.17.100
>>   Duration: 00:06:13.70, start: 0.00, bitrate: 5912 kb/s
>> Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v /
>> 0x7634706D), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 5777 kb/s, 29.97
>> fps, 29.97 tbr, 30k tbn, 30k tbc (default)
>> Metadata:
>>   handler_name: VideoHandler
>> Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz,
>> stereo, fltp, 128 kb/s (default)
>> Metadata:
>>   handler_name: SoundHandler
>> Stream mapping:
>>   Stream #0:0 -> #0:0 (mpeg4 (native) -> mpeg4 (native))
>>   Stream #0:1 -> #0:1 (aac (native) -> aac (native))
>> Press [q] to stop, [?] for help
>> free(): invalid pointer
>> Aborted (core dumped)
>>
>>
>>
>> When the output is an image, t does not crashes but neither fallback to
>> native
>>
>>
>> $ffmpeg -i in.jpg -vf srcnn=dnn_backend=tensorflow out.png
>>
>> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
>> developers
>>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>>   configuration:
>>   libavutil  56. 18.102 / 56. 18.102
>>   libavcodec 58. 19.105 / 58. 19.105
>>   libavformat58. 17.100 / 58. 17.100
>>   libavdevice58.  4.100 / 58.  4.100
>>   libavfilter 7. 25.100 /  7. 25.100
>>   libswscale  5.  2.100 /  5.  2.100
>>   libswresample   3.  2.100 /  3.  2.100
>> Input #0, image2, from 'in.jpg':
>>   Duration: 00:00:00.04, start: 0.00, bitrate: 43469 kb/s
>> Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown),
>> 1192x670 [SAR 1:1 DAR 596:335], 25 tbr, 25 tbn, 25 tbc
>> Stream mapping:
>>   Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))
>> Press [q] to stop, [?] for help
>> [Parsed_srcnn_0 @ 0x557d3ea55980] could not create DNN module for
>> requested backend
>> [AVFilterGraph @ 0x557d3ea102c0] Error initializing filter 'srcnn'
>> with args 'dnn_backend=tensorflow'
>> Error reinitializing filters!
>> Failed to inject frame into filter network: Cannot allocate memory
>> Error while processing the decoded data for stream #0:0
>> Conversion failed!
>>
>>
>> I think you could disable the tensorflow option if it is not enable in
>> configure or fallback to native, either solution is ok for me.
>
>
> I disabled tensorflow option when it is not configured with it. Here is the
> updated patch.
> I think, crash occurred due to improper call to av_freep for dnn_module.
> Here is also the patch, that fixes this bug.
>
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

2018-06-06 17:22 GMT+03:00 Pedro Arthur :

> Hi,
>
> 2018-06-05 20:23 GMT-03:00 Sergey Lavrushkin :
> > Here is the patch, that fixes described issues.
> When I try to run (video input), when tf is not enabled in configure it
> crashes.
>
>
> $ffmpeg -i in.mp4 -vf srcnn=dnn_backend=tensorflow out.mp4
>
> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
> developers
>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>   configuration:
>   libavutil  56. 18.102 / 56. 18.102
>   libavcodec 58. 19.105 / 58. 19.105
>   libavformat58. 17.100 / 58. 17.100
>   libavdevice58.  4.100 / 58.  4.100
>   libavfilter 7. 25.100 /  7. 25.100
>   libswscale  5.  2.100 /  5.  2.100
>   libswresample   3.  2.100 /  3.  2.100
> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mp4':
>   Metadata:
> major_brand : isom
> minor_version   : 512
> compatible_brands: isomiso2mp41
> encoder : Lavf58.17.100
>   Duration: 00:06:13.70, start: 0.00, bitrate: 5912 kb/s
> Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v /
> 0x7634706D), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 5777 kb/s, 29.97
> fps, 29.97 tbr, 30k tbn, 30k tbc (default)
> Metadata:
>   handler_name: VideoHandler
> Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz,
> stereo, fltp, 128 kb/s (default)
> Metadata:
>   handler_name: SoundHandler
> Stream mapping:
>   Stream #0:0 -> #0:0 (mpeg4 (native) -> mpeg4 (native))
>   Stream #0:1 -> #0:1 (aac (native) -> aac (native))
> Press [q] to stop, [?] for help
> free(): invalid pointer
> Aborted (core dumped)
>
>
>
> When the output is an image, t does not crashes but neither fallback to
> native
>
>
> $ffmpeg -i in.jpg -vf srcnn=dnn_backend=tensorflow out.png
>
> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
> developers
>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>   configuration:
>   libavutil  56. 18.102 / 56. 18.102
>   libavcodec 58. 19.105 / 58. 19.105
>   libavformat58. 17.100 / 58. 17.100
>   libavdevice58.  4.100 / 58.  4.100
>   libavfilter 7. 25.100 /  7. 25.100
>   libswscale  5.  2.100 /  5.  2.100
>   libswresample   3.  2.100 /  3.  2.100
> Input #0, image2, from 'in.jpg':
>   Duration: 00:00:00.04, start: 0.00, bitrate: 43469 kb/s
> Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown),
> 1192x670 [SAR 1:1 DAR 596:335], 25 tbr, 25 tbn, 25 tbc
> Stream mapping:
>   Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))
> Press [q] to stop, [?] for help
> [Parsed_srcnn_0 @ 0x557d3ea55980] could not create DNN module for
> requested backend
> [AVFilterGraph @ 0x557d3ea102c0] Error initializing filter 'srcnn'
> with args 'dnn_backend=tensorflow'
> Error reinitializing filters!
> Failed to inject frame into filter network: Cannot allocate memory
> Error while processing the decoded data for stream #0:0
> Conversion failed!
>
>
> I think you could disable the tensorflow option if it is not enable in
> configure or fallback to native, either solution is ok for me.


I disabled tensorflow option when it is not configured with it. Here is the
updated patch.
I think, crash occurred due to improper call to av_freep for dnn_module.
Here is also the patch, that fixes this bug.
From 33c1e08b650f3724c1317f024d716c8234e283b6 Mon Sep 17 00:00:00 2001
From: Sergey Lavrushkin 
Date: Wed, 6 Jun 2018 01:44:40 +0300
Subject: [PATCH 1/2] libavfilter/vf_srcnn.c: adds DNN module backend AVOption,
 changes AV_LOG_INFO message to AV_LOG_VERBOSE.

---
 libavfilter/vf_srcnn.c | 25 +++--
 1 file changed, 11 insertions(+), 14 deletions(-)

diff --git a/libavfilter/vf_srcnn.c b/libavfilter/vf_srcnn.c
index 5c5e26b33a..bba54f6780 100644
--- a/libavfilter/vf_srcnn.c
+++ b/libavfilter/vf_srcnn.c
@@ -36,6 +36,7 @@ typedef struct SRCNNContext {
 
 char* model_filename;
 float* input_output_buf;
+DNNBackendType backend_type;
 DNNModule* dnn_module;
 DNNModel* model;
 DNNData input_output;
@@ -44,6 +45,11 @@ typedef struct SRCNNContext {
 #define OFFSET(x) offsetof(SRCNNContext, x)
 #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
 static const AVOption srcnn_options[] = {
+{ "dnn_backend", "DNN backend used for model execution", OFFSET(backend_type), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, 1, FLAGS, "backend" },
+{ "native", "native backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "backend" },
+#if (CONFIG_LIBTENSORFLOW == 1)
+{ "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "backend" },
+#endif
 { "model_filename", "path to model file specifying network architecture and its parameters", OFFSET(model_filename), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS },
 { NULL }
 };
@@ -54,29 +60,20 @@ static av_cold int init(AVFilterContext* context)
 {
 SRCNNContext* srcnn_context = context->priv;
 
-srcnn_context->dnn_module = 

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

Hi,

2018-06-05 20:23 GMT-03:00 Sergey Lavrushkin :
> Here is the patch, that fixes described issues.
When I try to run (video input), when tf is not enabled in configure it crashes.


$ffmpeg -i in.mp4 -vf srcnn=dnn_backend=tensorflow out.mp4

ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration:
  libavutil  56. 18.102 / 56. 18.102
  libavcodec 58. 19.105 / 58. 19.105
  libavformat58. 17.100 / 58. 17.100
  libavdevice58.  4.100 / 58.  4.100
  libavfilter 7. 25.100 /  7. 25.100
  libswscale  5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mp4':
  Metadata:
major_brand : isom
minor_version   : 512
compatible_brands: isomiso2mp41
encoder : Lavf58.17.100
  Duration: 00:06:13.70, start: 0.00, bitrate: 5912 kb/s
Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v /
0x7634706D), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 5777 kb/s, 29.97
fps, 29.97 tbr, 30k tbn, 30k tbc (default)
Metadata:
  handler_name: VideoHandler
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz,
stereo, fltp, 128 kb/s (default)
Metadata:
  handler_name: SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> mpeg4 (native))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
free(): invalid pointer
Aborted (core dumped)



When the output is an image, t does not crashes but neither fallback to native


$ffmpeg -i in.jpg -vf srcnn=dnn_backend=tensorflow out.png

ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration:
  libavutil  56. 18.102 / 56. 18.102
  libavcodec 58. 19.105 / 58. 19.105
  libavformat58. 17.100 / 58. 17.100
  libavdevice58.  4.100 / 58.  4.100
  libavfilter 7. 25.100 /  7. 25.100
  libswscale  5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
Input #0, image2, from 'in.jpg':
  Duration: 00:00:00.04, start: 0.00, bitrate: 43469 kb/s
Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown),
1192x670 [SAR 1:1 DAR 596:335], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))
Press [q] to stop, [?] for help
[Parsed_srcnn_0 @ 0x557d3ea55980] could not create DNN module for
requested backend
[AVFilterGraph @ 0x557d3ea102c0] Error initializing filter 'srcnn'
with args 'dnn_backend=tensorflow'
Error reinitializing filters!
Failed to inject frame into filter network: Cannot allocate memory
Error while processing the decoded data for stream #0:0
Conversion failed!


I think you could disable the tensorflow option if it is not enable in
configure or fallback to native, either solution is ok for me.
>
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

2018-06-05 17:20 GMT+03:00 James Almer :

> On 6/3/2018 3:02 PM, Sergey Lavrushkin wrote:
> > diff --git a/libavfilter/vf_srcnn.c b/libavfilter/vf_srcnn.c
> > index d6efe9b478..5c5e26b33a 100644
> > --- a/libavfilter/vf_srcnn.c
> > +++ b/libavfilter/vf_srcnn.c
> > @@ -41,7 +41,6 @@ typedef struct SRCNNContext {
> >  DNNData input_output;
> >  } SRCNNContext;
> >
> > -
> >  #define OFFSET(x) offsetof(SRCNNContext, x)
> >  #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
> >  static const AVOption srcnn_options[] = {
> > @@ -55,10 +54,19 @@ static av_cold int init(AVFilterContext* context)
> >  {
> >  SRCNNContext* srcnn_context = context->priv;
> >
> > -srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
> > +srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF);
>
> This should be a filter AVOption, not hardcoded to one or another. What
> if i, for whatever reason, want to use the native backend when i have
> libtensorflow enabled?
>
> >  if (!srcnn_context->dnn_module){
> > -av_log(context, AV_LOG_ERROR, "could not create dnn module\n");
> > -return AVERROR(ENOMEM);
> > +srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
> > +if (!srcnn_context->dnn_module){
> > +av_log(context, AV_LOG_ERROR, "could not create dnn
> module\n");
> > +return AVERROR(ENOMEM);
> > +}
> > +else{
> > +av_log(context, AV_LOG_INFO, "using native backend for DNN
> inference\n");
>
> VERBOSE, not INFO
>
> > +}
> > +}
> > +else{
> > +av_log(context, AV_LOG_INFO, "using tensorflow backend for DNN
> inference\n");
>
> Ditto.
>
> >  }
> >  if (!srcnn_context->model_filename){
> >  av_log(context, AV_LOG_INFO, "model file for network was not
> specified, using default network for x2 upsampling\n");


Here is the patch, that fixes described issues.
From 971e15b4b1e3f2747aa07d0221f99226cba622ac Mon Sep 17 00:00:00 2001
From: Sergey Lavrushkin 
Date: Wed, 6 Jun 2018 01:44:40 +0300
Subject: [PATCH] libavfilter/vf_srcnn.c: adds DNN module backend AVOption,
 changes AV_LOG_INFO message to AV_LOG_VERBOSE.

---
 libavfilter/vf_srcnn.c | 23 +--
 1 file changed, 9 insertions(+), 14 deletions(-)

diff --git a/libavfilter/vf_srcnn.c b/libavfilter/vf_srcnn.c
index 5c5e26b33a..17e380503e 100644
--- a/libavfilter/vf_srcnn.c
+++ b/libavfilter/vf_srcnn.c
@@ -36,6 +36,7 @@ typedef struct SRCNNContext {
 
 char* model_filename;
 float* input_output_buf;
+DNNBackendType backend_type;
 DNNModule* dnn_module;
 DNNModel* model;
 DNNData input_output;
@@ -44,6 +45,9 @@ typedef struct SRCNNContext {
 #define OFFSET(x) offsetof(SRCNNContext, x)
 #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
 static const AVOption srcnn_options[] = {
+{ "dnn_backend", "DNN backend used for model execution", OFFSET(backend_type), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, 1, FLAGS, "backend" },
+{ "native", "native backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "backend" },
+{ "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "backend" },
 { "model_filename", "path to model file specifying network architecture and its parameters", OFFSET(model_filename), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS },
 { NULL }
 };
@@ -54,29 +58,20 @@ static av_cold int init(AVFilterContext* context)
 {
 SRCNNContext* srcnn_context = context->priv;
 
-srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF);
+srcnn_context->dnn_module = ff_get_dnn_module(srcnn_context->backend_type);
 if (!srcnn_context->dnn_module){
-srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
-if (!srcnn_context->dnn_module){
-av_log(context, AV_LOG_ERROR, "could not create dnn module\n");
-return AVERROR(ENOMEM);
-}
-else{
-av_log(context, AV_LOG_INFO, "using native backend for DNN inference\n");
-}
-}
-else{
-av_log(context, AV_LOG_INFO, "using tensorflow backend for DNN inference\n");
+av_log(context, AV_LOG_ERROR, "could not create DNN module for requested backend\n");
+return AVERROR(ENOMEM);
 }
 if (!srcnn_context->model_filename){
-av_log(context, AV_LOG_INFO, "model file for network was not specified, using default network for x2 upsampling\n");
+av_log(context, AV_LOG_VERBOSE, "model file for network was not specified, using default network for x2 upsampling\n");
 srcnn_context->model = (srcnn_context->dnn_module->load_default_model)(DNN_SRCNN);
 }
 else{
 srcnn_context->model = (srcnn_context->dnn_module->load_model)(srcnn_context->model_filename);
 }
 if (!srcnn_context->model){
-av_log(context, AV_LOG_ERROR, "could not load dnn model\n");
+av_log(context, AV_LOG_ERROR, "could not load DNN model\n");
 

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[James Almer]

On 6/3/2018 3:02 PM, Sergey Lavrushkin wrote:
> diff --git a/libavfilter/vf_srcnn.c b/libavfilter/vf_srcnn.c
> index d6efe9b478..5c5e26b33a 100644
> --- a/libavfilter/vf_srcnn.c
> +++ b/libavfilter/vf_srcnn.c
> @@ -41,7 +41,6 @@ typedef struct SRCNNContext {
>  DNNData input_output;
>  } SRCNNContext;
>  
> -
>  #define OFFSET(x) offsetof(SRCNNContext, x)
>  #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
>  static const AVOption srcnn_options[] = {
> @@ -55,10 +54,19 @@ static av_cold int init(AVFilterContext* context)
>  {
>  SRCNNContext* srcnn_context = context->priv;
>  
> -srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
> +srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF);

This should be a filter AVOption, not hardcoded to one or another. What
if i, for whatever reason, want to use the native backend when i have
libtensorflow enabled?

>  if (!srcnn_context->dnn_module){
> -av_log(context, AV_LOG_ERROR, "could not create dnn module\n");
> -return AVERROR(ENOMEM);
> +srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
> +if (!srcnn_context->dnn_module){
> +av_log(context, AV_LOG_ERROR, "could not create dnn module\n");
> +return AVERROR(ENOMEM);
> +}
> +else{
> +av_log(context, AV_LOG_INFO, "using native backend for DNN 
> inference\n");

VERBOSE, not INFO

> +}
> +}
> +else{
> +av_log(context, AV_LOG_INFO, "using tensorflow backend for DNN 
> inference\n");

Ditto.

>  }
>  if (!srcnn_context->model_filename){
>  av_log(context, AV_LOG_INFO, "model file for network was not 
> specified, using default network for x2 upsampling\n");

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

Pushed.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

2018-06-03 17:05 GMT-03:00 Pedro Arthur :
> 2018-06-03 16:55 GMT-03:00 Sergey Lavrushkin :
>>> My concern is when we add more models, currently we have to store 2
>>> models, one for the "native" implementation and one for the TF
>>> backend.
>>> There is also the case were one wants to update the weights for a
>>> model, it will be necessary to update both the native and TF data.
>>> Having duplicated data is much easier to get inconsistencies between
>>> implementations.
>>
>>
>> I understand the problem, but I am afraid that manual graph construction can
>> take a lot of time, especially if we add something more complicated than
>> srcnn,
>> and the second approach passing weights in placeholders will require to
>> add some logic for it in other parts of API besides model loading.
>> I am thinking of another way, that is to get weights for native model from
>> this binary
>> tf model, if they are stored there consistently, and not specify them as
>> float arrays.
>> But then for each new model we need to find offsets for each weights array.
> Ok, we can keep it until we find a reasonable solution.


If there are nothing else to be addressed I'll push it by tomorrow.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

2018-06-03 16:55 GMT-03:00 Sergey Lavrushkin :
>> My concern is when we add more models, currently we have to store 2
>> models, one for the "native" implementation and one for the TF
>> backend.
>> There is also the case were one wants to update the weights for a
>> model, it will be necessary to update both the native and TF data.
>> Having duplicated data is much easier to get inconsistencies between
>> implementations.
>
>
> I understand the problem, but I am afraid that manual graph construction can
> take a lot of time, especially if we add something more complicated than
> srcnn,
> and the second approach passing weights in placeholders will require to
> add some logic for it in other parts of API besides model loading.
> I am thinking of another way, that is to get weights for native model from
> this binary
> tf model, if they are stored there consistently, and not specify them as
> float arrays.
> But then for each new model we need to find offsets for each weights array.
Ok, we can keep it until we find a reasonable solution.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

>
> My concern is when we add more models, currently we have to store 2
> models, one for the "native" implementation and one for the TF
> backend.
> There is also the case were one wants to update the weights for a
> model, it will be necessary to update both the native and TF data.
> Having duplicated data is much easier to get inconsistencies between
> implementations.
>

I understand the problem, but I am afraid that manual graph construction can
take a lot of time, especially if we add something more complicated than
srcnn,
and the second approach passing weights in placeholders will require to
add some logic for it in other parts of API besides model loading.
I am thinking of another way, that is to get weights for native model from
this binary
tf model, if they are stored there consistently, and not specify them as
float arrays.
But then for each new model we need to find offsets for each weights array.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

2018-06-03 15:25 GMT-03:00 Sergey Lavrushkin :
> 2018-06-03 19:57 GMT+03:00 Pedro Arthur :
>>
>> 2018-05-31 12:01 GMT-03:00 Sergey Lavrushkin :
>> > Hello,
>> >
>> > This patch introduces TensorFlow backend for DNN inference module.
>> > This backend uses TensorFlow binary models and requires from model
>> > to have the operation named 'x' as an input operation and the operation
>> > named 'y' as an output operation. Models are executed using
>> > libtensorflow.
>>
>> Hi,
>>
>> You added the tf model in dnn_srcnn.h, it seems the data is being
>> duplicated as it already contains the weights as C float arrays.
>> Is it possible to construct the model graph via C api and set the
>> weights using the ones we already have, eliminating the need for
>> storing the whole tf model?
>
>
> Hi,
>
> I think, it is possible, but it will require to manually create every
> operation
> and specify each of their attributes and inputs in a certain order specified
> by
> operations declaration. Here is that model:
> https://drive.google.com/file/d/1s7bW7QnUfmTaYoMLPdYYTOLujqNgRq0J/view?usp=sharing
> It is just a lot easier to store the whole model and not construct it
> manually.
> Another way, I think of, is to pass weights in placeholders and not save
> them in
> model, but it has to be done when session is already created and not during
> model
> loading. Maybe some init operation can be specified with variables
> assignment to values
> passed through placeholders during model loading, if it is possible. But is
> it really crucial
> to not store the whole tf model? It is not that big.

My concern is when we add more models, currently we have to store 2
models, one for the "native" implementation and one for the TF
backend.
There is also the case were one wants to update the weights for a
model, it will be necessary to update both the native and TF data.
Having duplicated data is much easier to get inconsistencies between
implementations.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

2018-06-03 19:57 GMT+03:00 Pedro Arthur :

> 2018-05-31 12:01 GMT-03:00 Sergey Lavrushkin :
> > Hello,
> >
> > This patch introduces TensorFlow backend for DNN inference module.
> > This backend uses TensorFlow binary models and requires from model
> > to have the operation named 'x' as an input operation and the operation
> > named 'y' as an output operation. Models are executed using
> libtensorflow.
>
> Hi,
>
> You added the tf model in dnn_srcnn.h, it seems the data is being
> duplicated as it already contains the weights as C float arrays.
> Is it possible to construct the model graph via C api and set the
> weights using the ones we already have, eliminating the need for
> storing the whole tf model?


Hi,

I think, it is possible, but it will require to manually create every
operation
and specify each of their attributes and inputs in a certain order
specified by
operations declaration. Here is that model:
https://drive.google.com/file/d/1s7bW7QnUfmTaYoMLPdYYTOLujqNgRq0J/view?usp=sharing
It is just a lot easier to store the whole model and not construct it
manually.
Another way, I think of, is to pass weights in placeholders and not save
them in
model, but it has to be done when session is already created and not during
model
loading. Maybe some init operation can be specified with variables
assignment to values
passed through placeholders during model loading, if it is possible. But is
it really crucial
to not store the whole tf model? It is not that big.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Pedro Arthur]

2018-05-31 12:01 GMT-03:00 Sergey Lavrushkin :
> Hello,
>
> This patch introduces TensorFlow backend for DNN inference module.
> This backend uses TensorFlow binary models and requires from model
> to have the operation named 'x' as an input operation and the operation
> named 'y' as an output operation. Models are executed using libtensorflow.

Hi,

You added the tf model in dnn_srcnn.h, it seems the data is being
duplicated as it already contains the weights as C float arrays.
Is it possible to construct the model graph via C api and set the
weights using the ones we already have, eliminating the need for
storing the whole tf model?
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[James Almer]

On 6/2/2018 3:45 PM, Sergey Lavrushkin wrote:
> 2018-06-02 19:45 GMT+03:00 James Almer  >:
> 
> On 5/31/2018 12:01 PM, Sergey Lavrushkin wrote:
> > diff --git a/Changelog b/Changelog
> > index df2024fb59..a667fd045d 100644
> > --- a/Changelog
> > +++ b/Changelog
> > @@ -11,6 +11,7 @@ version :
> >  - support mbedTLS based TLS
> >  - DNN inference interface
> >  - Reimplemented SRCNN filter using DNN inference interface
> > +- TensorFlow DNN backend
> 
> This and the two entries you added earlier don't really belong here.
> It's enough with the line stating the filter was introduced back in
> ffmpeg 4.0
> 
> 
> I should not add any line regarding introduced DNN inference module,
> that can be usefull for someone writing another filter based on DNN?

Changelog lists the newly added features of interest to users, not
developers. When the CBS framework was added it wasn't mentioned here
either, only the filters making use of it.

Users don't care that the scrnn filter was reimplemented using this new
framework, but are however interested to know that the filter was added,
or at most if its features were considerably expanded.
So based on that, I guess it's acceptable to add a line in this patch like

- libtensorflow backend for DNN based filters like srcnn.

Or similar. But the earlier two entries don't belong here.

>  
> 
> > 
> > 
> >  version 4.0:
> > diff --git a/configure b/configure
> > index 09ff0c55e2..47e21fec39 100755
> > --- a/configure
> > +++ b/configure
> > @@ -259,6 +259,7 @@ External library support:
> >    --enable-libspeex        enable Speex de/encoding via libspeex [no]
> >    --enable-libsrt          enable Haivision SRT protocol via
> libsrt [no]
> >    --enable-libssh          enable SFTP protocol via libssh [no]
> > +  --enable-libtensorflow   enable TensorFlow as a DNN module
> backend [no]
> 
> Maybe mention it's for the srcnn filter.
> 
> >    --enable-libtesseract    enable Tesseract, needed for ocr
> filter [no]
> >    --enable-libtheora       enable Theora encoding via libtheora [no]
> >    --enable-libtls          enable LibreSSL (via libtls), needed
> for https support
> > @@ -1713,6 +1714,7 @@ EXTERNAL_LIBRARY_LIST="
> >      libspeex
> >      libsrt
> >      libssh
> > +    libtensorflow
> >      libtesseract
> >      libtheora
> >      libtwolame
> > @@ -3453,7 +3455,7 @@ avcodec_select="null_bsf"
> >  avdevice_deps="avformat avcodec avutil"
> >  avdevice_suggest="libm"
> >  avfilter_deps="avutil"
> > -avfilter_suggest="libm"
> > +avfilter_suggest="libm libtensorflow"
> 
> Add instead
> 
> srcnn_filter_suggest="libtensorflow"
> 
> To the corresponding section.
> 
> 
> But this DNN inference module can be used for other filters.
> At least, I think, that after training more complicated models for super
> resolution I'll have to add them as separate filters.
> So, I thought, this module shouldn't be a part of srcnn filter from the
> begining.
> Or is it better to add  *_filter_suggest="libtensorflow" to the
> configure script and  
> dnn_*.o to the Makefile for every new filter based on this module? 

Yes. If it's a non public framework/API used by one or more modules that
can be disabled (as is the case with filters) then it should be added to
the list of object dependencies for each of those optional modules. See
framesync.o for another example of this.

That being said, another way to implement this is making DNN its own
independent module in configure, like for example cbs, fft and many
others. This way, configure would look something like

dnn_suggest="libtensorflow"
srcnn_filter_select="dnn"

With dnn being an entry added to the CONFIG_EXTRA list.

Then in libavfilter/Makefile you'd do

OBJS-$(CONFIG_DNN) += dnn_interface.o dnn_backend_native.o dnn_backend_tf.o

Or even better, something like

DNN-OBJS-$(CONFIG_LIBTENSORFLOW) += dnn_backend_tf.o
OBJS-$(CONFIG_DNN) += dnn_interface.o dnn_backend_native.o $(DNN-OBJS-yes)

Which would allow you to remove the #if preprocessor check in
dnn_backend_tf.c, as it will only be compiled when both libtensorflow
and at least one filter using dnn are enabled.

The reason for all this is to have everything as modular as possible,
reducing compiled objects and dependencies (like lavfi linking to lavf)
to exactly what's strictly needed.

>  
> 
> >  avformat_deps="avcodec avutil"
> >  avformat_suggest="libm network zlib"
> >  avresample_deps="avutil"
> > @@ -6055,6 +6057,7 @@ enabled libsoxr           && require libsoxr
> soxr.h soxr_create -lsoxr
> >  enabled libssh            && require_pkg_config libssh libssh
> libssh/sftp.h sftp_init
> >  enabled libspeex          && require_pkg_config libspeex speex
> speex/speex.h speex_decoder_init
> >  enabled libsrt            && 

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

2018-06-02 19:45 GMT+03:00 James Almer :

> On 5/31/2018 12:01 PM, Sergey Lavrushkin wrote:
> > diff --git a/Changelog b/Changelog
> > index df2024fb59..a667fd045d 100644
> > --- a/Changelog
> > +++ b/Changelog
> > @@ -11,6 +11,7 @@ version :
> >  - support mbedTLS based TLS
> >  - DNN inference interface
> >  - Reimplemented SRCNN filter using DNN inference interface
> > +- TensorFlow DNN backend
>
> This and the two entries you added earlier don't really belong here.
> It's enough with the line stating the filter was introduced back in
> ffmpeg 4.0
>

I should not add any line regarding introduced DNN inference module,
that can be usefull for someone writing another filter based on DNN?


> >
> >
> >  version 4.0:
> > diff --git a/configure b/configure
> > index 09ff0c55e2..47e21fec39 100755
> > --- a/configure
> > +++ b/configure
> > @@ -259,6 +259,7 @@ External library support:
> >--enable-libspeexenable Speex de/encoding via libspeex [no]
> >--enable-libsrt  enable Haivision SRT protocol via libsrt [no]
> >--enable-libssh  enable SFTP protocol via libssh [no]
> > +  --enable-libtensorflow   enable TensorFlow as a DNN module backend
> [no]
>
> Maybe mention it's for the srcnn filter.
>
> >--enable-libtesseractenable Tesseract, needed for ocr filter [no]
> >--enable-libtheora   enable Theora encoding via libtheora [no]
> >--enable-libtls  enable LibreSSL (via libtls), needed for
> https support
> > @@ -1713,6 +1714,7 @@ EXTERNAL_LIBRARY_LIST="
> >  libspeex
> >  libsrt
> >  libssh
> > +libtensorflow
> >  libtesseract
> >  libtheora
> >  libtwolame
> > @@ -3453,7 +3455,7 @@ avcodec_select="null_bsf"
> >  avdevice_deps="avformat avcodec avutil"
> >  avdevice_suggest="libm"
> >  avfilter_deps="avutil"
> > -avfilter_suggest="libm"
> > +avfilter_suggest="libm libtensorflow"
>
> Add instead
>
> srcnn_filter_suggest="libtensorflow"
>
> To the corresponding section.
>

But this DNN inference module can be used for other filters.
At least, I think, that after training more complicated models for super
resolution I'll have to add them as separate filters.
So, I thought, this module shouldn't be a part of srcnn filter from the
begining.
Or is it better to add  *_filter_suggest="libtensorflow" to the configure
script and
dnn_*.o to the Makefile for every new filter based on this module?


> >  avformat_deps="avcodec avutil"
> >  avformat_suggest="libm network zlib"
> >  avresample_deps="avutil"
> > @@ -6055,6 +6057,7 @@ enabled libsoxr   && require libsoxr
> soxr.h soxr_create -lsoxr
> >  enabled libssh&& require_pkg_config libssh libssh
> libssh/sftp.h sftp_init
> >  enabled libspeex  && require_pkg_config libspeex speex
> speex/speex.h speex_decoder_init
> >  enabled libsrt&& require_pkg_config libsrt "srt >= 1.2.0"
> srt/srt.h srt_socket
> > +enabled libtensorflow && require libtensorflow tensorflow/c/c_api.h
> TF_Version -ltensorflow && add_cflags -DTENSORFLOW_BACKEND
>
> Superfluous define. Just check for CONFIG_LIBTENSORFLOW instead.
>
> >  enabled libtesseract  && require_pkg_config libtesseract tesseract
> tesseract/capi.h TessBaseAPICreate
> >  enabled libtheora && require libtheora theora/theoraenc.h
> th_info_init -ltheoraenc -ltheoradec -logg
> >  enabled libtls&& require_pkg_config libtls libtls tls.h
> tls_configure
> > diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> > index 3201cbeacf..82915e2f75 100644
> > --- a/libavfilter/Makefile
> > +++ b/libavfilter/Makefile
> > @@ -14,6 +14,7 @@ OBJS = allfilters.o
>  \
> > buffersrc.o
> \
> > dnn_interface.o
> \
> > dnn_backend_native.o
>  \
> > +   dnn_backend_tf.o
>  \
>
> See Jan Ekström's patch. Add this to the filter's entry as all these
> source files should not be compiled unconditionally.
>
> > drawutils.o
> \
> > fifo.o
>  \
> > formats.o
> \
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[James Almer]

On 5/31/2018 12:01 PM, Sergey Lavrushkin wrote:
> diff --git a/Changelog b/Changelog
> index df2024fb59..a667fd045d 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -11,6 +11,7 @@ version :
>  - support mbedTLS based TLS
>  - DNN inference interface
>  - Reimplemented SRCNN filter using DNN inference interface
> +- TensorFlow DNN backend

This and the two entries you added earlier don't really belong here.
It's enough with the line stating the filter was introduced back in
ffmpeg 4.0

>  
>  
>  version 4.0:
> diff --git a/configure b/configure
> index 09ff0c55e2..47e21fec39 100755
> --- a/configure
> +++ b/configure
> @@ -259,6 +259,7 @@ External library support:
>--enable-libspeexenable Speex de/encoding via libspeex [no]
>--enable-libsrt  enable Haivision SRT protocol via libsrt [no]
>--enable-libssh  enable SFTP protocol via libssh [no]
> +  --enable-libtensorflow   enable TensorFlow as a DNN module backend [no]

Maybe mention it's for the srcnn filter.

>--enable-libtesseractenable Tesseract, needed for ocr filter [no]
>--enable-libtheora   enable Theora encoding via libtheora [no]
>--enable-libtls  enable LibreSSL (via libtls), needed for https 
> support
> @@ -1713,6 +1714,7 @@ EXTERNAL_LIBRARY_LIST="
>  libspeex
>  libsrt
>  libssh
> +libtensorflow
>  libtesseract
>  libtheora
>  libtwolame
> @@ -3453,7 +3455,7 @@ avcodec_select="null_bsf"
>  avdevice_deps="avformat avcodec avutil"
>  avdevice_suggest="libm"
>  avfilter_deps="avutil"
> -avfilter_suggest="libm"
> +avfilter_suggest="libm libtensorflow"

Add instead

srcnn_filter_suggest="libtensorflow"

To the corresponding section.

>  avformat_deps="avcodec avutil"
>  avformat_suggest="libm network zlib"
>  avresample_deps="avutil"
> @@ -6055,6 +6057,7 @@ enabled libsoxr   && require libsoxr soxr.h 
> soxr_create -lsoxr
>  enabled libssh&& require_pkg_config libssh libssh libssh/sftp.h 
> sftp_init
>  enabled libspeex  && require_pkg_config libspeex speex speex/speex.h 
> speex_decoder_init
>  enabled libsrt&& require_pkg_config libsrt "srt >= 1.2.0" 
> srt/srt.h srt_socket
> +enabled libtensorflow && require libtensorflow tensorflow/c/c_api.h 
> TF_Version -ltensorflow && add_cflags -DTENSORFLOW_BACKEND

Superfluous define. Just check for CONFIG_LIBTENSORFLOW instead.

>  enabled libtesseract  && require_pkg_config libtesseract tesseract 
> tesseract/capi.h TessBaseAPICreate
>  enabled libtheora && require libtheora theora/theoraenc.h 
> th_info_init -ltheoraenc -ltheoradec -logg
>  enabled libtls&& require_pkg_config libtls libtls tls.h 
> tls_configure
> diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> index 3201cbeacf..82915e2f75 100644
> --- a/libavfilter/Makefile
> +++ b/libavfilter/Makefile
> @@ -14,6 +14,7 @@ OBJS = allfilters.o 
> \
> buffersrc.o  \
> dnn_interface.o  \
> dnn_backend_native.o \
> +   dnn_backend_tf.o \

See Jan Ekström's patch. Add this to the filter's entry as all these
source files should not be compiled unconditionally.

> drawutils.o  \
> fifo.o   \
> formats.o\


___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

2018-06-01 6:09 GMT+03:00 Guo, Yejun :

> Did you try to build ffmpeg with TENSORFLOW_BACKEND enabled, and run it
> without TF library?  This case is possible when an end user install
> pre-built package on a machine without TF library.
>
> In function init, the logic is to fall back to cpu path (DNN_NATIVE) if
> unable to load tensorflow backend. While in function ff_get_dnn_module, it
> has no chance to 'return NULL'.
>

I tried to run ffmpeg built with libtensorflow enabled and without
tensorflow library, it didn't start. I got this message:

ffmpeg: error while loading shared libraries: libtensorflow.so: cannot open
shared object file: No such file or directory

Is it even possible to run it without library that was enabled during
configuration? Maybe I need to change something in the
configure script? Otherwise there is no point to add any fallback to
DNN_NATIVE, if it just won't start.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Guo, Yejun]

Did you try to build ffmpeg with TENSORFLOW_BACKEND enabled, and run it without 
TF library?  This case is possible when an end user install pre-built package 
on a machine without TF library.

In function init, the logic is to fall back to cpu path (DNN_NATIVE) if unable 
to load tensorflow backend. While in function ff_get_dnn_module, it has no 
chance to 'return NULL'.

static av_cold int init(AVFilterContext* context)
{
SRCNNContext* srcnn_context = context->priv;

#ifdef TENSORFLOW_BACKEND
srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF);
if (!srcnn_context->dnn_module){
av_log(context, AV_LOG_INFO, "could not load tensorflow backend, using 
native backend instead\n");
srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
}
#else
srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
#endif



DNNModule* ff_get_dnn_module(DNNBackendType backend_type)
{
...
case DNN_TF:
#ifdef TENSORFLOW_BACKEND
//add a runtime check here, possible?
dnn_module->load_model = _dnn_load_model_tf;
dnn_module->load_default_model = _dnn_load_default_model_tf;
dnn_module->execute_model = _dnn_execute_model_tf;
dnn_module->free_model = _dnn_free_model_tf;
#else
av_freep(dnn_module);
return NULL;
#endif



Thanks
Yejun (郭叶军)

-Original Message-
From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of Sergey 
Lavrushkin
Sent: Thursday, May 31, 2018 11:01 PM
To: FFmpeg development discussions and patches 
Cc: Pedro Arthur 
Subject: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN 
module

Hello,

This patch introduces TensorFlow backend for DNN inference module.
This backend uses TensorFlow binary models and requires from model to have the 
operation named 'x' as an input operation and the operation named 'y' as an 
output operation. Models are executed using libtensorflow.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel