Re: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for avg_pool
> -Original Message-
> From: ffmpeg-devel On Behalf Of Guo,
> Yejun
> Sent: Thursday, July 30, 2020 10:02 AM
> To: FFmpeg development discussions and patches
> Subject: Re: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for
> avg_pool
>
>
>
> > -Original Message-
> > From: ffmpeg-devel On Behalf Of Fu,
> > Ting
> > Sent: 2020年7月30日 9:43
> > To: FFmpeg development discussions and patches
> >
> > Subject: Re: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native
> > support for avg_pool
> >
> >
> >
> > > -Original Message-
> > > From: ffmpeg-devel On Behalf Of
> > > Ting Fu
> > > Sent: Wednesday, July 29, 2020 10:11 PM
> > > To: ffmpeg-devel@ffmpeg.org
> > > Subject: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native
> > > support for avg_pool
> > >
> > > Not support pooling strides in channel dimension now.
> > > It can be tested with the model generated with below python script:
> > >
> > > import tensorflow as tf
> > > import numpy as np
> > > import imageio
> > >
> > > in_img = imageio.imread('input_odd.jpg') in_img =
> > > in_img.astype(np.float32)/255.0 in_data = in_img[np.newaxis, :]
> > >
> > > x = tf.placeholder(tf.float32, shape=[1, None, None, 3],
> > > name='dnn_in') x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1],
> > > strides=[1,2,2,1], padding='SAME') #please alter the params as
> > > needed y = tf.identity(x_pool, name='dnn_out')
> > >
> > > sess=tf.Session()
> > > sess.run(tf.global_variables_initializer())
> > >
> > > graph_def = tf.graph_util.convert_variables_to_constants(sess,
> > > sess.graph_def,
> > > ['dnn_out']) tf.train.write_graph(graph_def, '.',
> > > 'image_process.pb',
> > > as_text=False)
> > >
> > > print("image_process.pb generated, please use \
> > > path_to_ffmpeg/tools/python/convert.py to generate
> > > image_process.model\n")
> > >
> > > output = sess.run(y, feed_dict={x: in_data})
> > > imageio.imsave("out.jpg",
> > > np.squeeze(output))
> > >
> > > Signed-off-by: Ting Fu
> > > ---
> > > libavfilter/dnn/Makefile | 1 +
> > > libavfilter/dnn/dnn_backend_native.h | 2 +
> > > .../dnn/dnn_backend_native_layer_avgpool.c| 147
> > ++
> > > .../dnn/dnn_backend_native_layer_avgpool.h| 35 +
> > > .../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
> > > libavfilter/dnn/dnn_backend_native_layers.c | 2 +
> > > tools/python/convert_from_tensorflow.py | 35 -
> > > 7 files changed, 222 insertions(+), 3 deletions(-) create mode
> > > 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
> > > create mode 100644
> > > libavfilter/dnn/dnn_backend_native_layer_avgpool.h
> > >
> > > diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
> > > index d90137ec42..e0957073ee 100644
> > > --- a/libavfilter/dnn/Makefile
> > > +++ b/libavfilter/dnn/Makefile
> > > @@ -1,6 +1,7 @@
> > > OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_interface.o
> > > OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_backend_native.o
> > > OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_backend_native_layers.o
> > > +OBJS-$(CONFIG_DNN) +=
> > > dnn/dnn_backend_native_layer_avgpool.o
> > > OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_backend_native_layer_pad.o
> > > OBJS-$(CONFIG_DNN) +=
> > > dnn/dnn_backend_native_layer_conv2d.o
> > > OBJS-$(CONFIG_DNN) +=
> > > dnn/dnn_backend_native_layer_depth2space.o
> > [...]
> > >
> > >
> > > +def dump_avg_pool_to_file(self, node, f):
> > > +assert(node.op == 'AvgPool')
> > > +self.layer_number = self.layer_number + 1
> > > +self.converted_nodes.add(node.name)
> > > +node0 = self.name_node_dict[node.input[0]]
> > > +strides = node.attr['strides']
> > > +assert(strides.list.i[1]==strides.list.i[2])
> > > +assert(strides.list.i[0]==1)
> > > +assert(strides.list.i[3]==1)
> >
> > Since the tensorflow do not support pooling strides in batch
> > dimension, and current do not support pooling in channel dimension, added
> two assert here.
>
> thanks, and please add the comments within the code.
Thank you Yejun,
if no further comments, this would be the only difference in patch V3.
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org
> with subject "unsubscribe".
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for avg_pool
> -Original Message-
> From: ffmpeg-devel On Behalf Of Fu, Ting
> Sent: 2020年7月30日 9:43
> To: FFmpeg development discussions and patches
> Subject: Re: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for
> avg_pool
>
>
>
> > -Original Message-
> > From: ffmpeg-devel On Behalf Of Ting
> > Fu
> > Sent: Wednesday, July 29, 2020 10:11 PM
> > To: ffmpeg-devel@ffmpeg.org
> > Subject: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support
> > for avg_pool
> >
> > Not support pooling strides in channel dimension now.
> > It can be tested with the model generated with below python script:
> >
> > import tensorflow as tf
> > import numpy as np
> > import imageio
> >
> > in_img = imageio.imread('input_odd.jpg') in_img =
> > in_img.astype(np.float32)/255.0 in_data = in_img[np.newaxis, :]
> >
> > x = tf.placeholder(tf.float32, shape=[1, None, None, 3],
> > name='dnn_in') x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1],
> > strides=[1,2,2,1], padding='SAME') #please alter the params as needed
> > y = tf.identity(x_pool, name='dnn_out')
> >
> > sess=tf.Session()
> > sess.run(tf.global_variables_initializer())
> >
> > graph_def = tf.graph_util.convert_variables_to_constants(sess,
> > sess.graph_def,
> > ['dnn_out']) tf.train.write_graph(graph_def, '.', 'image_process.pb',
> > as_text=False)
> >
> > print("image_process.pb generated, please use \
> > path_to_ffmpeg/tools/python/convert.py to generate
> > image_process.model\n")
> >
> > output = sess.run(y, feed_dict={x: in_data}) imageio.imsave("out.jpg",
> > np.squeeze(output))
> >
> > Signed-off-by: Ting Fu
> > ---
> > libavfilter/dnn/Makefile | 1 +
> > libavfilter/dnn/dnn_backend_native.h | 2 +
> > .../dnn/dnn_backend_native_layer_avgpool.c| 147
> ++
> > .../dnn/dnn_backend_native_layer_avgpool.h| 35 +
> > .../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
> > libavfilter/dnn/dnn_backend_native_layers.c | 2 +
> > tools/python/convert_from_tensorflow.py | 35 -
> > 7 files changed, 222 insertions(+), 3 deletions(-) create mode
> > 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
> > create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
> >
> > diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile index
> > d90137ec42..e0957073ee 100644
> > --- a/libavfilter/dnn/Makefile
> > +++ b/libavfilter/dnn/Makefile
> > @@ -1,6 +1,7 @@
> > OBJS-$(CONFIG_DNN) +=
> dnn/dnn_interface.o
> > OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native.o
> > OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layers.o
> > +OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_backend_native_layer_avgpool.o
> > OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layer_pad.o
> > OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_backend_native_layer_conv2d.o
> > OBJS-$(CONFIG_DNN) +=
> > dnn/dnn_backend_native_layer_depth2space.o
> [...]
> >
> >
> > +def dump_avg_pool_to_file(self, node, f):
> > +assert(node.op == 'AvgPool')
> > +self.layer_number = self.layer_number + 1
> > +self.converted_nodes.add(node.name)
> > +node0 = self.name_node_dict[node.input[0]]
> > +strides = node.attr['strides']
> > +assert(strides.list.i[1]==strides.list.i[2])
> > +assert(strides.list.i[0]==1)
> > +assert(strides.list.i[3]==1)
>
> Since the tensorflow do not support pooling strides in batch dimension, and
> current do not support pooling in channel dimension, added two assert here.
thanks, and please add the comments within the code.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for avg_pool
> -Original Message-
> From: ffmpeg-devel On Behalf Of Ting Fu
> Sent: Wednesday, July 29, 2020 10:11 PM
> To: ffmpeg-devel@ffmpeg.org
> Subject: [FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for
> avg_pool
>
> Not support pooling strides in channel dimension now.
> It can be tested with the model generated with below python script:
>
> import tensorflow as tf
> import numpy as np
> import imageio
>
> in_img = imageio.imread('input_odd.jpg') in_img =
> in_img.astype(np.float32)/255.0 in_data = in_img[np.newaxis, :]
>
> x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in') x_pool
> = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
> #please
> alter the params as needed y = tf.identity(x_pool, name='dnn_out')
>
> sess=tf.Session()
> sess.run(tf.global_variables_initializer())
>
> graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
> ['dnn_out']) tf.train.write_graph(graph_def, '.', 'image_process.pb',
> as_text=False)
>
> print("image_process.pb generated, please use \
> path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
>
> output = sess.run(y, feed_dict={x: in_data}) imageio.imsave("out.jpg",
> np.squeeze(output))
>
> Signed-off-by: Ting Fu
> ---
> libavfilter/dnn/Makefile | 1 +
> libavfilter/dnn/dnn_backend_native.h | 2 +
> .../dnn/dnn_backend_native_layer_avgpool.c| 147 ++
> .../dnn/dnn_backend_native_layer_avgpool.h| 35 +
> .../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
> libavfilter/dnn/dnn_backend_native_layers.c | 2 +
> tools/python/convert_from_tensorflow.py | 35 -
> 7 files changed, 222 insertions(+), 3 deletions(-) create mode 100644
> libavfilter/dnn/dnn_backend_native_layer_avgpool.c
> create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
>
> diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile index
> d90137ec42..e0957073ee 100644
> --- a/libavfilter/dnn/Makefile
> +++ b/libavfilter/dnn/Makefile
> @@ -1,6 +1,7 @@
> OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
> OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
> OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layers.o
> +OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layer_avgpool.o
> OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layer_pad.o
> OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layer_conv2d.o
> OBJS-$(CONFIG_DNN) +=
> dnn/dnn_backend_native_layer_depth2space.o
[...]
>
>
> +def dump_avg_pool_to_file(self, node, f):
> +assert(node.op == 'AvgPool')
> +self.layer_number = self.layer_number + 1
> +self.converted_nodes.add(node.name)
> +node0 = self.name_node_dict[node.input[0]]
> +strides = node.attr['strides']
> +assert(strides.list.i[1]==strides.list.i[2])
> +assert(strides.list.i[0]==1)
> +assert(strides.list.i[3]==1)
Since the tensorflow do not support pooling strides in batch dimension, and
current do not support pooling in channel dimension,
added two assert here.
> +strides = strides.list.i[1]
> +filter_node = node.attr['ksize']
> +input_name = node.input[0]
> +
> +assert(filter_node.list.i[0]==1)
> +assert(filter_node.list.i[3]==1)
Same as above, the tensorflow do not support pooling ksize in both batch
dimension and channel dimension.
> +filter_height = filter_node.list.i[1]
> +filter_width = filter_node.list.i[2]
> +
> +in_channels = node0.attr['shape'].shape.dim[3].size
> +out_channels = in_channels
> +padding = node.attr['padding'].s.decode("utf-8")
> +np.array([self.op2code[node.op], strides,
> self.pool_paddings[padding],
> in_channels, out_channels,
> + filter_height],dtype=np.uint32).tofile(f)
> +
> +input_operand_index = self.add_operand(input_name,
> Operand.IOTYPE_INPUT)
> +output_operand_index = self.add_operand(node.name,
> Operand.IOTYPE_OUTPUT)
> +np.array([input_operand_index,
> + output_operand_index],dtype=np.uint32).tofile(f)
> +
> +
> def dump_layers_to_file(self, f):
> for node in self.nodes:
> if node.name in self.converted_nodes:
> @@ -311,6 +342,8 @@ class TFConverter:
>
> if node.op == 'Conv2D':
> self.dump_simple_conv2d_to_file(node, f)
> +if node.op ==
[FFmpeg-devel] [PATCH V2 1/2] dnn/native: add native support for avg_pool
Not support pooling strides in channel dimension now.
It can be tested with the model generated with below python script:
import tensorflow as tf
import numpy as np
import imageio
in_img = imageio.imread('input_odd.jpg')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]
x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
x_pool = tf.nn.avg_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')
#please alter the params as needed
y = tf.identity(x_pool, name='dnn_out')
sess=tf.Session()
sess.run(tf.global_variables_initializer())
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def,
['dnn_out'])
tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False)
print("image_process.pb generated, please use \
path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n")
output = sess.run(y, feed_dict={x: in_data})
imageio.imsave("out.jpg", np.squeeze(output))
Signed-off-by: Ting Fu
---
libavfilter/dnn/Makefile | 1 +
libavfilter/dnn/dnn_backend_native.h | 2 +
.../dnn/dnn_backend_native_layer_avgpool.c| 147 ++
.../dnn/dnn_backend_native_layer_avgpool.h| 35 +
.../dnn/dnn_backend_native_layer_conv2d.h | 3 +-
libavfilter/dnn/dnn_backend_native_layers.c | 2 +
tools/python/convert_from_tensorflow.py | 35 -
7 files changed, 222 insertions(+), 3 deletions(-)
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.c
create mode 100644 libavfilter/dnn/dnn_backend_native_layer_avgpool.h
diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
index d90137ec42..e0957073ee 100644
--- a/libavfilter/dnn/Makefile
+++ b/libavfilter/dnn/Makefile
@@ -1,6 +1,7 @@
OBJS-$(CONFIG_DNN) += dnn/dnn_interface.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native.o
OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layers.o
+OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_avgpool.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_pad.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_conv2d.o
OBJS-$(CONFIG_DNN) +=
dnn/dnn_backend_native_layer_depth2space.o
diff --git a/libavfilter/dnn/dnn_backend_native.h
b/libavfilter/dnn/dnn_backend_native.h
index 62191ffe88..26e9a33387 100644
--- a/libavfilter/dnn/dnn_backend_native.h
+++ b/libavfilter/dnn/dnn_backend_native.h
@@ -43,10 +43,12 @@ typedef enum {
DLT_MAXIMUM = 4,
DLT_MATH_BINARY = 5,
DLT_MATH_UNARY = 6,
+DLT_AVG_POOL = 7,
DLT_COUNT
} DNNLayerType;
typedef enum {DOT_INPUT = 1, DOT_OUTPUT = 2, DOT_INTERMEDIATE = DOT_INPUT |
DOT_OUTPUT} DNNOperandType;
+typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNPaddingParam;
typedef struct Layer{
DNNLayerType type;
diff --git a/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
new file mode 100644
index 00..a6ebb0db8f
--- /dev/null
+++ b/libavfilter/dnn/dnn_backend_native_layer_avgpool.c
@@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2020
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * DNN native backend implementation.
+ */
+
+#include "libavutil/avassert.h"
+#include "dnn_backend_native_layer_avgpool.h"
+
+int dnn_load_layer_avg_pool(Layer *layer, AVIOContext *model_file_context, int
file_size, int operands_num)
+{
+AvgPoolParams *avgpool_params;
+int dnn_size = 0;
+avgpool_params = av_malloc(sizeof(*avgpool_params));
+if(!avgpool_params)
+return 0;
+
+avgpool_params->strides = (int32_t)avio_rl32(model_file_context);
+avgpool_params->padding_method = (int32_t)avio_rl32(model_file_context);
+avgpool_params->in_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->out_channels = (int32_t)avio_rl32(model_file_context);
+avgpool_params->kernel_size = (int32_t)avio_rl32(model_file_context);
+dnn_size += 20;
+
+if (dnn_size > file_size || avgpool_params->in_channels <= 0 ||
+avgpool_params->out_channels <= 0 || avgpool_params->kernel_size <= 0
||
+
