[FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping.
An example command to use this filter with vaapi codecs:
FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \
opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \
vaapi -i INPUT -filter_hw_device ocl -filter_complex \
'[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \
[x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 OUTPUT
Signed-off-by: Ruiling Song
---
As I didn't receive any other comment on v3, this version only fix the comment
from Michael.
And also include some little change to leverage CL_SET_KERNEL_ARG() macro.
Thanks!
Ruiling
configure | 1 +
libavfilter/Makefile| 2 +
libavfilter/allfilters.c| 1 +
libavfilter/colorspace.c| 90 +
libavfilter/colorspace.h| 41 +++
libavfilter/opencl/colorspace_common.cl | 220 +++
libavfilter/opencl/tonemap.cl | 272 ++
libavfilter/opencl_source.h | 2 +
libavfilter/vf_tonemap_opencl.c | 624
9 files changed, 1253 insertions(+)
create mode 100644 libavfilter/colorspace.c
create mode 100644 libavfilter/colorspace.h
create mode 100644 libavfilter/opencl/colorspace_common.cl
create mode 100644 libavfilter/opencl/tonemap.cl
create mode 100644 libavfilter/vf_tonemap_opencl.c
diff --git a/configure b/configure
index 333e326..d9c5d63 100755
--- a/configure
+++ b/configure
@@ -3411,6 +3411,7 @@ tinterlace_filter_deps="gpl"
tinterlace_merge_test_deps="tinterlace_filter"
tinterlace_pad_test_deps="tinterlace_filter"
tonemap_filter_deps="const_nan"
+tonemap_opencl_filter_deps="opencl const_nan"
unsharp_opencl_filter_deps="opencl"
uspp_filter_deps="gpl avcodec"
vaguedenoiser_filter_deps="gpl"
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index 5b4be49..d2c85cf 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -356,6 +356,8 @@ OBJS-$(CONFIG_TINTERLACE_FILTER) +=
vf_tinterlace.o
OBJS-$(CONFIG_TLUT2_FILTER) += vf_lut2.o framesync.o
OBJS-$(CONFIG_TMIX_FILTER) += vf_mix.o framesync.o
OBJS-$(CONFIG_TONEMAP_FILTER)+= vf_tonemap.o
+OBJS-$(CONFIG_TONEMAP_OPENCL_FILTER) += vf_tonemap_opencl.o
colorspace.o opencl.o \
+opencl/tonemap.o
opencl/colorspace_common.o
OBJS-$(CONFIG_TRANSPOSE_FILTER) += vf_transpose.o
OBJS-$(CONFIG_TRIM_FILTER) += trim.o
OBJS-$(CONFIG_UNPREMULTIPLY_FILTER) += vf_premultiply.o framesync.o
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
index f2d27d2..fa85c29 100644
--- a/libavfilter/allfilters.c
+++ b/libavfilter/allfilters.c
@@ -345,6 +345,7 @@ extern AVFilter ff_vf_tinterlace;
extern AVFilter ff_vf_tlut2;
extern AVFilter ff_vf_tmix;
extern AVFilter ff_vf_tonemap;
+extern AVFilter ff_vf_tonemap_opencl;
extern AVFilter ff_vf_transpose;
extern AVFilter ff_vf_trim;
extern AVFilter ff_vf_unpremultiply;
diff --git a/libavfilter/colorspace.c b/libavfilter/colorspace.c
new file mode 100644
index 000..7fd7bdf
--- /dev/null
+++ b/libavfilter/colorspace.c
@@ -0,0 +1,90 @@
+/*
+ * Copyright (c) 2016 Ronald S. Bultje
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include "colorspace.h"
+
+
+void invert_matrix3x3(const double in[3][3], double out[3][3])
+{
+double m00 = in[0][0], m01 = in[0][1], m02 = in[0][2],
+ m10 = in[1][0], m11 = in[1][1], m12 = in[1][2],
+ m20 = in[2][0], m21 = in[2][1], m22 = in[2][2];
+int i, j;
+double det;
+
+out[0][0] = (m11 * m22 - m21 * m12);
+out[0][1] = -(m01 * m22 - m21 * m02);
+out[0][2] = (m01 * m12 - m11 * m02);
+out[1][0] = -(m10 * m22 - m20 * m12);
+out[1][1] = (m00 * m22 - m20 * m02);
+out[1][2] = -(m00 * m12 - m10 * m02);
+out[2][0] = (m10 * m21 - m20 * m11);
+out[2][1] = -(m00 * m21 - m20 * m01);
+out[2][2] = (m00 * m11 - m10 * m01);
+
+det = m00 * out[0][0] + m10 * out[0][1] + m20 * out[0][2];
+det = 1.0 / det;
+
+for (i = 0; i < 3; i++) {
+for (j = 0; j <
Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
On Tue, Jun 19, 2018 at 09:57:31 +0800, Ruiling Song wrote: > As I didn't receive any other comment on v3, this version only fix the > comment from Michael. > And also include some little change to leverage CL_SET_KERNEL_ARG() macro. Could you kindly add some documentation to doc/filters.texi, at least describing the filter's options? > This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping. > > An example command to use this filter with vaapi codecs: > FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \ > opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \ > vaapi -i INPUT -filter_hw_device ocl -filter_complex \ > '[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \ > [x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 OUTPUT This could also go into the documentation. Thanks, Moritz ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
On 19/06/18 02:57, Ruiling Song wrote: > This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping. > > An example command to use this filter with vaapi codecs: > FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \ > opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \ > vaapi -i INPUT -filter_hw_device ocl -filter_complex \ > '[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \ > [x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 OUTPUT > > Signed-off-by: Ruiling Song > --- > As I didn't receive any other comment on v3, this version only fix the > comment from Michael. > And also include some little change to leverage CL_SET_KERNEL_ARG() macro. > > Thanks! > Ruiling > > configure | 1 + > libavfilter/Makefile| 2 + > libavfilter/allfilters.c| 1 + > libavfilter/colorspace.c| 90 + > libavfilter/colorspace.h| 41 +++ > libavfilter/opencl/colorspace_common.cl | 220 +++ > libavfilter/opencl/tonemap.cl | 272 ++ > libavfilter/opencl_source.h | 2 + > libavfilter/vf_tonemap_opencl.c | 624 > > 9 files changed, 1253 insertions(+) > create mode 100644 libavfilter/colorspace.c > create mode 100644 libavfilter/colorspace.h > create mode 100644 libavfilter/opencl/colorspace_common.cl > create mode 100644 libavfilter/opencl/tonemap.cl > create mode 100644 libavfilter/vf_tonemap_opencl.c I did a bit more testing, LGTM; therefore applied. Some further thoughts: * Did you get anywhere with testing on other platforms? (I'm happy with the current state with it working on at least two different platforms including the actually-useful GPU-with-interop ones, but it would be nice to know what was going wrong in the other cases.) * The single-frame-delay effect ends up looking a bit weird when I go looking for it - I now keep seeing the flash of a single frame at a different brightness on some transitions, though I'm not sure it's obvious enough that I would notice often if I didn't already know it was there. How much performance would it actually cost to use the correct frame rather than the previous one? * +1 to the comment from Moritz about documentation if you wouldn't mind writing some as a separate patch. Thanks, - Mark ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Moritz Barsnick > Sent: Wednesday, June 20, 2018 7:32 PM > To: FFmpeg development discussions and patches > Subject: Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter. > > On Tue, Jun 19, 2018 at 09:57:31 +0800, Ruiling Song wrote: > > As I didn't receive any other comment on v3, this version only fix the > > comment > from Michael. > > And also include some little change to leverage CL_SET_KERNEL_ARG() macro. > > Could you kindly add some documentation to doc/filters.texi, at least > describing the filter's options? > > > This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping. > > > > An example command to use this filter with vaapi codecs: > > FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \ > > opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \ > > vaapi -i INPUT -filter_hw_device ocl -filter_complex \ > > '[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \ > > [x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 > OUTPUT > > This could also go into the documentation. Yes, thanks. Although I am not quite good at English writing, I will try to add some document and send out for review. Thanks! Ruiling > > Thanks, > Moritz > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
On Thu, Jun 21, 2018 at 8:33 AM Mark Thompson wrote: > > On 19/06/18 02:57, Ruiling Song wrote: > > This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping. > > > > An example command to use this filter with vaapi codecs: > > FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \ > > opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \ > > vaapi -i INPUT -filter_hw_device ocl -filter_complex \ > > '[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \ > > [x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 OUTPUT > > > > Signed-off-by: Ruiling Song > > --- > > As I didn't receive any other comment on v3, this version only fix the > > comment from Michael. > > And also include some little change to leverage CL_SET_KERNEL_ARG() macro. > > > > Thanks! > > Ruiling > > > > configure | 1 + > > libavfilter/Makefile| 2 + > > libavfilter/allfilters.c| 1 + > > libavfilter/colorspace.c| 90 + > > libavfilter/colorspace.h| 41 +++ > > libavfilter/opencl/colorspace_common.cl | 220 +++ > > libavfilter/opencl/tonemap.cl | 272 ++ > > libavfilter/opencl_source.h | 2 + > > libavfilter/vf_tonemap_opencl.c | 624 > > > > 9 files changed, 1253 insertions(+) > > create mode 100644 libavfilter/colorspace.c > > create mode 100644 libavfilter/colorspace.h > > create mode 100644 libavfilter/opencl/colorspace_common.cl > > create mode 100644 libavfilter/opencl/tonemap.cl > > create mode 100644 libavfilter/vf_tonemap_opencl.c > > I did a bit more testing, LGTM; therefore applied. > > Some further thoughts: > * Did you get anywhere with testing on other platforms? (I'm happy with the > current state with it working on at least two different platforms including > the actually-useful GPU-with-interop ones, but it would be nice to know what > was going wrong in the other cases.) Now I start to run intel NEO OpenCL GPU driver (now available on github (https://github.com/intel/compute-runtime).) + iHD open source driver(https://github.com/intel/media-driver), any status will update to the mail list. > * The single-frame-delay effect ends up looking a bit weird when I go looking > for it - I now keep seeing the flash of a single frame at a different > brightness on some transitions, though I'm not sure it's obvious enough that > I would often notice if I didn't already know it was there. How much > performance would it actually cost to use the correct frame rather than the > previous one? > * +1 to the comment from Moritz about documentation if you wouldn't mind > writing some as a separate patch. > +1 for comment, now OpenCL AVFilter use the complex command option, the other thing is if someone uses the FFmpeg API to enable the OpenCL AVFilter, I think we need some sample code in doc/examples to demo the API enable the OpenCL HWaccel. > Thanks, > > - Mark > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
> -Original Message- > From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of > Mark Thompson > Sent: Thursday, June 21, 2018 8:33 AM > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter. > > On 19/06/18 02:57, Ruiling Song wrote: > > This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping. > > > > An example command to use this filter with vaapi codecs: > > FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \ > > opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \ > > vaapi -i INPUT -filter_hw_device ocl -filter_complex \ > > '[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \ > > [x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 > OUTPUT > > > > Signed-off-by: Ruiling Song > > --- > > As I didn't receive any other comment on v3, this version only fix the > > comment > from Michael. > > And also include some little change to leverage CL_SET_KERNEL_ARG() macro. > > > > Thanks! > > Ruiling > > > > configure | 1 + > > libavfilter/Makefile| 2 + > > libavfilter/allfilters.c| 1 + > > libavfilter/colorspace.c| 90 + > > libavfilter/colorspace.h| 41 +++ > > libavfilter/opencl/colorspace_common.cl | 220 +++ > > libavfilter/opencl/tonemap.cl | 272 ++ > > libavfilter/opencl_source.h | 2 + > > libavfilter/vf_tonemap_opencl.c | 624 > > > 9 files changed, 1253 insertions(+) > > create mode 100644 libavfilter/colorspace.c > > create mode 100644 libavfilter/colorspace.h > > create mode 100644 libavfilter/opencl/colorspace_common.cl > > create mode 100644 libavfilter/opencl/tonemap.cl > > create mode 100644 libavfilter/vf_tonemap_opencl.c > > I did a bit more testing, LGTM; therefore applied. > > Some further thoughts: > * Did you get anywhere with testing on other platforms? (I'm happy with the > current state with it working on at least two different platforms including > the > actually-useful GPU-with-interop ones, but it would be nice to know what was > going wrong in the other cases.) So, this version with segfault fix still does not work on Mali platform, same error as before? For the other platform, do you mean other hardware? Currently I don't have other hardware to use. But I will add support for yuv420p so we can make it work with pocl. > * The single-frame-delay effect ends up looking a bit weird when I go looking > for > it - I now keep seeing the flash of a single frame at a different brightness > on > some transitions, though I'm not sure it's obvious enough that I would notice > often if I didn't already know it was there. How much performance would it > actually cost to use the correct frame rather than the previous one? I will try to remove this one frame delay. But it need some time. I think it is not easy to predict how much performance effect it would be. May be not too much. I will continue improve this filter step by step. > * +1 to the comment from Moritz about documentation if you wouldn't mind > writing some as a separate patch. Will add it. > > Thanks, > > - Mark > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter.
On 21/06/18 07:03, Song, Ruiling wrote: >> -Original Message- >> From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of >> Mark Thompson >> Sent: Thursday, June 21, 2018 8:33 AM >> To: ffmpeg-devel@ffmpeg.org >> Subject: Re: [FFmpeg-devel] [PATCH v4 1/2] lavfi: add opencl tonemap filter. >> >> On 19/06/18 02:57, Ruiling Song wrote: >>> This filter does HDR(HDR10/HLG) to SDR conversion with tone-mapping. >>> >>> An example command to use this filter with vaapi codecs: >>> FFMPEG -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device \ >>> opencl=ocl@va -hwaccel vaapi -hwaccel_device va -hwaccel_output_format \ >>> vaapi -i INPUT -filter_hw_device ocl -filter_complex \ >>> '[0:v]hwmap,tonemap_opencl=t=bt2020:tonemap=linear:format=p010[x1]; \ >>> [x1]hwmap=derive_device=vaapi:reverse=1' -c:v hevc_vaapi -profile 2 >> OUTPUT >>> >>> Signed-off-by: Ruiling Song >>> --- >>> As I didn't receive any other comment on v3, this version only fix the >>> comment >> from Michael. >>> And also include some little change to leverage CL_SET_KERNEL_ARG() macro. >>> >>> Thanks! >>> Ruiling >>> >>> configure | 1 + >>> libavfilter/Makefile| 2 + >>> libavfilter/allfilters.c| 1 + >>> libavfilter/colorspace.c| 90 + >>> libavfilter/colorspace.h| 41 +++ >>> libavfilter/opencl/colorspace_common.cl | 220 +++ >>> libavfilter/opencl/tonemap.cl | 272 ++ >>> libavfilter/opencl_source.h | 2 + >>> libavfilter/vf_tonemap_opencl.c | 624 >> >>> 9 files changed, 1253 insertions(+) >>> create mode 100644 libavfilter/colorspace.c >>> create mode 100644 libavfilter/colorspace.h >>> create mode 100644 libavfilter/opencl/colorspace_common.cl >>> create mode 100644 libavfilter/opencl/tonemap.cl >>> create mode 100644 libavfilter/vf_tonemap_opencl.c >> >> I did a bit more testing, LGTM; therefore applied. >> >> Some further thoughts: >> * Did you get anywhere with testing on other platforms? (I'm happy with the >> current state with it working on at least two different platforms including >> the >> actually-useful GPU-with-interop ones, but it would be nice to know what was >> going wrong in the other cases.) > So, this version with segfault fix still does not work on Mali platform, same > error as before? Complete log below, running on a RK3288 with Mali T760. (The input file here is from <http://4kmedia.org/lg-new-york-hdr-uhd-4k-demo/>, but it doesn't appear to matter.) Thanks, - Mark ffmpeg started on 2018-07-02 at 22:27:59 Report written to "ffmpeg-20180702-222759.log" Command line: ./ffmpeg_g -report -v 55 -y -threads 1 -i "/home/mrt/test/LG New York HDR UHD 4K Demo.ts" -init_hw_device opencl -filter_hw_device opencl0 -an -vf "format=p010,hwupload,tonemap_opencl=format=nv12,hwdownload,format=nv12" -c:v libx264 out.mp4 ffmpeg version N-91405-g54b425a7fa Copyright (c) 2000-2018 the FFmpeg developers built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516 configuration: --enable-debug --enable-opencl --enable-libdrm --enable-rkmpp --enable-gpl --enable-version3 --enable-libx264 --extra-ldflags='-L/usr/local/lib -lmali-midgard-r13p0-fbdev' libavutil 56. 18.102 / 56. 18.102 libavcodec 58. 20.104 / 58. 20.104 libavformat58. 17.101 / 58. 17.101 libavdevice58. 4.101 / 58. 4.101 libavfilter 7. 25.100 / 7. 25.100 libswscale 5. 2.100 / 5. 2.100 libswresample 3. 2.100 / 3. 2.100 libpostproc55. 2.100 / 55. 2.100 Splitting the commandline. Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'. Reading option '-v' ... matched as option 'v' (set logging level) with argument '55'. Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'. Reading option '-threads' ... matched as AVOption 'threads' with argument '1'. Reading option '-i' ... matched as input url with argument '/home/mrt/test/LG New York HDR UHD 4K Demo.ts'. Reading option '-init_hw_device' ... matched as option 'init_hw_device' (initialise hardware device) with argument 'opencl'. Reading option '-filter_hw_device' ... matched as option
