Re: Building libav/ffmpeg & x264 on 5.4
Hello, Actually, you greatly piqued my interest right there. So I have started fooling around with the ports Makefiles of graphics/ffmpeg and multimedia/x264 until i finally managed to pull it off! Needed some modifications of ffmpegs Makefile to throw out all dependencies on libx264 and also modifications of x264s Makefile, patches etc. But I got it working, proving that using modified ports you can easily reverse the ffmpeg/x264 linking. The only thing that remains is the need for a newer GNU assembler to make x264 work at proper speeds. You can leave it out, but that'd cripple x264 in any case. Even if you built ffmpeg+x264 from the stock ports, H.264/AVC encoding speed will be crippled when encoding with ffmpeg. Crippled x264 (or libx264) is just no good. So to make H.264/AVC encoding work properly, the GNU binutils / gas dependency remains in any case. Since there are no x86 binutils packages or ports, I still had to build that from the source code at "ftp://sourceware.org/pub/binutils/snapshots";. But, cool stuff. I never thought of using modified ports and "reversing" the way how x264 and ffmpeg link against each other. On 03/28/2014 11:04 AM, Dmitrij D. Czarkoff wrote: Michael Lackner said: Hello, And that's why I cannot use ffmpeg directly. Part of what I am doing is bringing a certain x264 benchmark onto a wide variety of systems, and to do that properly, I would need to use the exact same options for comparisons sake. After managing, proper documentation is being provided online. You may simply copy ffmpeg and x264 ports to "mystuff" directory inside your ports dir and make the changes you need to these "local" ports. That would allow you to reuse ports build infrastructure (which does many things easier) while having your customization.
Re: Building libav/ffmpeg & x264 on 5.4
Michael Lackner said: > Hello, > > And that's why I cannot use ffmpeg directly. Part of what I am doing is > bringing a certain > x264 benchmark onto a wide variety of systems, and to do that properly, I > would need to > use the exact same options for comparisons sake. After managing, proper > documentation is > being provided online. You may simply copy ffmpeg and x264 ports to "mystuff" directory inside your ports dir and make the changes you need to these "local" ports. That would allow you to reuse ports build infrastructure (which does many things easier) while having your customization. -- Dmitrij D. Czarkoff
Re: Building libav/ffmpeg & x264 on 5.4
And me again, I tried that now, setting CFLAGS/CXXFLAGS/LDFLAGS to "-I/usr/local/include -L/usr/local/lib", and then linking the current x264 release against OpenBSDs ffmpeg package, but it makes no difference. x264's config.log says: checking for sws_init_context(0,0,0); in libswscale/swscale.h... yes checking whether defined(AV_PIX_FMT_FLAG_RGB) is true... no -- conftext.c:3:2: error: #error -- Failed program was: -- #include #if !(defined(AV_PIX_FMT_FLAG_RGB)) #error #endif -- Also, I made sure that I was using GCC 4.2.1 and not the newer 4.8.1 here, so that x264 and OpenBSDs ffmpeg would "fit together". No go. When linking against my homebrewn libav (or ffmpeg!) it would just say this: checking whether defined(AV_PIX_FMT_FLAG_RGB) is true... yes And it works. I presume this has something to do with the age difference, as OpenBSDs ffmpeg and my x264 are almost a year apart, and that's usually too much to have them link against each other as by my previous experience. Maybe it's also because of the way ffmpeg was built and linked against libx264? It seems the OpenBSD solution (which reminds me of Debian Linux btw., which does it the same way) encourages the user to run the ffmpeg cli binary instead of using the x264 cli binary. I guess this is just the way most people do it. Use libx264 to encode H.264 by calling the ffmpeg binary and its libs to do the decoding part. The way I do it: I'm using the ffmpeg or libav libraries to do the decoding, and the x264 binary to do the encoding. Either way, my problem can be solved, and now I know how anyway. :) On 03/27/2014 09:17 AM, Stuart Henderson wrote: On 2014-03-26, Michael Lackner wrote: As for the other posts: Installing ffmpeg as a package can be done, but x264 would refuse to link against it. Could that be something as simple as needing to include -L/usr/local/lib in LDFLAGS when configuring x264? -- Michael Lackner Lehrstuhl für Informationstechnologie (CiT) Montanuniversität Leoben Tel.: +43 (0)3842/402-1505 | Mail: michael.lack...@unileoben.ac.at Fax.: +43 (0)3842/402-1502 | Web: http://institute.unileoben.ac.at/infotech
Re: Building libav/ffmpeg & x264 on 5.4
Hello, And that's why I cannot use ffmpeg directly. Part of what I am doing is bringing a certain x264 benchmark onto a wide variety of systems, and to do that properly, I would need to use the exact same options for comparisons sake. After managing, proper documentation is being provided online. Also, that way the same benchmarking script can be provided to users as-is. Maybe these days the options I've been using can be mapped to ffmpeg 100%, I haven't checked that out yet. I will however try what Mr. Henderson said and set my include/library paths to /usr/local/include and /usr/local/lib, just like I needed to do for GNU binutils 2.24 too, and then attempt to link x264 against OpenBSDs ffmpeg package again. But I doubt it'll work, as the x264 configure/build scripts do check /usr/local/ by default. But I can try. Besides, being stubborn and trying to get this to work "my" way at all costs made me learn a bunch of stuff about OpenBSD (and other systems too) along the way, and that can't be a bad thing, right? ;) Best, Michael On 03/27/2014 10:47 PM, Arnaud Bergeron wrote: Since ffmpeg is linked against x264 you could just use it to read and encode stuff to mp4. There is a guide[1] about how to do 2 pass, use presets and the rest. You can also use -x264opts to pass options directly to libx264 (but it doesn't support everything). [1] https://trac.ffmpeg.org/wiki/x264EncodingGuide 2014-03-27 4:17 GMT-04:00 Stuart Henderson : On 2014-03-26, Michael Lackner wrote: As for the other posts: Installing ffmpeg as a package can be done, but x264 would refuse to link against it. Could that be something as simple as needing to include -L/usr/local/lib in LDFLAGS when configuring x264?
Re: Building libav/ffmpeg & x264 on 5.4
Since ffmpeg is linked against x264 you could just use it to read and encode stuff to mp4. There is a guide[1] about how to do 2 pass, use presets and the rest. You can also use -x264opts to pass options directly to libx264 (but it doesn't support everything). [1] https://trac.ffmpeg.org/wiki/x264EncodingGuide 2014-03-27 4:17 GMT-04:00 Stuart Henderson : > On 2014-03-26, Michael Lackner wrote: > > As for the other posts: Installing ffmpeg as a package can be done, but > x264 would refuse > > to link against it. > > Could that be something as simple as needing to include -L/usr/local/lib > in LDFLAGS when configuring x264?
Re: Building libav/ffmpeg & x264 on 5.4
On 2014-03-26, Michael Lackner wrote: > As for the other posts: Installing ffmpeg as a package can be done, but x264 > would refuse > to link against it. Could that be something as simple as needing to include -L/usr/local/lib in LDFLAGS when configuring x264?
Re: Building libav/ffmpeg & x264 on 5.4
Good morning,
Well, holy crap. I'm feeling a bit stupid now, but I am still very new to
OpenBSD. I
wasn't aware of all the limits the system would impose on a regular user. It is
pretty
nice though, from a security and stability perspective.
So, to "fix" this, I did "chpass " as root and added my user to the
"staff" login
class. Then I edited "/etc/login.conf" to increase the default amount of memory
that a
user belonging to the staff class can allocate. From 512MB to 2048MB. The
process I am
running might require something just shy of or slightly above 1GB, so yeah. I
was just
too lazy to create my own class, so "staff" will have to do.
And, well. My build works now. That was exactly why I needed some input from
experienced
people, because I just don't have an idea of how to REALLY work with OpenBSD
(yet).
So thanks for that!
As for the other posts: Installing ffmpeg as a package can be done, but x264
would refuse
to link against it. Maybe the newest x264 source code is just too new or
something? Or
some headers are missing? No idea. So I went back to building libav myself and
link x264
against that.
Also it's nice to hear that 5.5 will come with a more modern assembler, I have
played
around with LLVM/Clang before, didn't know Clang included its own assembler
too. Looking
forward to the next release, and thanks again! :)
Best,
Michael
On 03/25/2014 04:32 PM, Andrei Vrincianu wrote:
Hi Michael,
Maybe it's not because of this, but did you try raising the data segment
size limit for your user? ulimit -a should help.
Best,
Andrei Vrincianu
On Tue, Mar 25, 2014 at 3:35 PM, Michael Lackner <
michael.lack...@unileoben.ac.at> wrote:
Greetings!
This is my first question here, please be gentle! ;)
So, as the subject line says, I am currently attempting to compile,
install and run
either the libav or ffmpeg media codec suite and then x264 on OpenBSD 5.4,
where x264
is then linked against either libav or ffmpeg. All of this on the
x86_64/AMD64
architecture. x264 itself is a H.264/AVC video trans-/encoder.
So far so good.
The first big problem was the lack of a new enough GNU assembler
("as"/"gas"), as x264
features SSSE3 inline assembly, that gas from binutils 2.15 cannot build.
So I went ahead
and compiled myself my own gas from binutils 2.24, which supposedly worked
fine.
But the real bummer is what follows, and this error even shows up, if I
disable all
assembly optimizations in libav/ffmpeg as well as x264 (then it even
compiles on a stock
5.4 without my new gas 2.24, but won't run).
So, I start my video transcoding, and I get this (leaving out the [info]
lines):
"x264 [error]: malloc of size 8856384 failed
x264 [error]: x264_encoder_encode failed
aborted at input frame 13, output frame 0"
I suspect x264 and not the libav/ffmpeg it was linked against, because 13
input frames
were seemingly decoded by libav/ffmpeg, but not a single output frame was
encoded by x264.
Using older versions of x264/libav (up to 1 year old I tried) results in
the same problem,
only the malloc() size number is different. Like "31736" instead of the
"8856384" above.
Just in case you'd wanna know, this is a sample command line like the one
I called:
"x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f
-2:0 --bitrate 1 --aq-mode 1 -p 1 --slow-firstpass --stats
framestats.stats -t 2 --no-fast-pskip --cqm flat input.264 -o
pass1output.264"
I tried compiling this with and without assembly, and with both GCC 4.2.1
as well as GCC
4.8.1. The error is always the same.
To learn more, I thought I'd take a look at the x264 port of OpenBSD 5.4,
but found this
in its Makefile, disabling all linking against both libav and ffmpeg as
well as disabling
all assembly (likely due to the binutils/gas issue):
CONFIGURE_ARGS+=--disable-asm \
--disable-ffms \
--disable-gpac \
--disable-lavf \
--disable-swscale \
--enable-static \
--prefix=${PREFIX}
The ffms thing disables linking against ffmpeg, the lavf+swscale stuff
disables linking
against libav. asm is self-explanatory.
So the x264 port (as well as the precompiled package) are completely
crippled. Not only is
the assembly missing, costing tons of performance, but you can't even feed
anything but
raw video to it! What I need is the capability to feed stuff like
H.264/AVC, MPEG2, VC-1
videostreams etc. to x264, so I need libav or ffmpeg.
Now, the main issue is the malloc() failure here. My home-brewn gas
shouldn't be the
problem, because it happens even when compiling from pure C/C++.
My assumption would be, that maybe OpenBSDs libc implementation of
malloc() behaves in
ways that x264 can't handle properly?! I've tried looking at the x264
source coude, but
this stuff is just way beyond me, I don't understand any of the code
really.
I have so far managed to do this on NetBSD, FreeBSD/PC-BSD, MidnightBSD,
Dragonfly BSD,
OpenSolaris, Linux, Windows (CygWin and MinGW/MSYS), MacOS X and even
Haiku OS with
varying degrees of modifications
Re: Building libav/ffmpeg & x264 on 5.4
On 2014-03-25, Michael Lackner wrote:
> The first big problem was the lack of a new enough GNU assembler
> ("as"/"gas"), as x264
> features SSSE3 inline assembly, that gas from binutils 2.15 cannot build. So
> I went ahead
> and compiled myself my own gas from binutils 2.24, which supposedly worked
> fine.
OpenBSD -current (and the already-tagged but not-yet-released 5.5) builds
x264 with Clang's assembler to enable the assembly code.
> To learn more, I thought I'd take a look at the x264 port of OpenBSD 5.4, but
> found this
> in its Makefile, disabling all linking against both libav and ffmpeg as well
> as disabling
> all assembly (likely due to the binutils/gas issue):
The FFmpeg port is built against x264, so building x264 against FFmpeg would
give a circular build dependency. It could potentially be done differently by
adding a bootstrap flavour of the x264 port, but that's a lot of complication
for something which I don't recall seeing any requests on-list for before.
Re: Building libav/ffmpeg & x264 on 5.4
On Tue, Mar 25, 2014 at 6:35 AM, Michael Lackner wrote: > So, as the subject line says, I am currently attempting to compile, install > and run > either the libav or ffmpeg media codec suite and then x264 on OpenBSD 5.4, > where x264 is then linked against either libav or ffmpeg. All of this on the > x86_64/AMD64 architecture. x264 itself is a H.264/AVC video trans-/encoder. The good news is that the ports team has already done the work to make these compile and run on OpenBSD. Take a look at http://www.openbsd.org/faq/faq15.html You should be able to just set the PKG_PATH environment variable to point to a packages ftp site for version 5.4 and do "pkg_add ffmpeg", for example. If you have some reason to redo that work, then you should at least review the patches in the ports tree to see how they did the porting. Philip Guenther
Re: Building libav/ffmpeg & x264 on 5.4
Hi Michael,
Maybe it's not because of this, but did you try raising the data segment
size limit for your user? ulimit -a should help.
Best,
Andrei Vrincianu
On Tue, Mar 25, 2014 at 3:35 PM, Michael Lackner <
michael.lack...@unileoben.ac.at> wrote:
> Greetings!
>
> This is my first question here, please be gentle! ;)
>
> So, as the subject line says, I am currently attempting to compile,
> install and run
> either the libav or ffmpeg media codec suite and then x264 on OpenBSD 5.4,
> where x264
> is then linked against either libav or ffmpeg. All of this on the
> x86_64/AMD64
> architecture. x264 itself is a H.264/AVC video trans-/encoder.
>
> So far so good.
>
> The first big problem was the lack of a new enough GNU assembler
> ("as"/"gas"), as x264
> features SSSE3 inline assembly, that gas from binutils 2.15 cannot build.
> So I went ahead
> and compiled myself my own gas from binutils 2.24, which supposedly worked
> fine.
>
> But the real bummer is what follows, and this error even shows up, if I
> disable all
> assembly optimizations in libav/ffmpeg as well as x264 (then it even
> compiles on a stock
> 5.4 without my new gas 2.24, but won't run).
>
> So, I start my video transcoding, and I get this (leaving out the [info]
> lines):
>
>
> "x264 [error]: malloc of size 8856384 failed
> x264 [error]: x264_encoder_encode failed
> aborted at input frame 13, output frame 0"
>
>
> I suspect x264 and not the libav/ffmpeg it was linked against, because 13
> input frames
> were seemingly decoded by libav/ffmpeg, but not a single output frame was
> encoded by x264.
>
> Using older versions of x264/libav (up to 1 year old I tried) results in
> the same problem,
> only the malloc() size number is different. Like "31736" instead of the
> "8856384" above.
>
> Just in case you'd wanna know, this is a sample command line like the one
> I called:
>
>
> "x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f
> -2:0 --bitrate 1 --aq-mode 1 -p 1 --slow-firstpass --stats
> framestats.stats -t 2 --no-fast-pskip --cqm flat input.264 -o
> pass1output.264"
>
>
> I tried compiling this with and without assembly, and with both GCC 4.2.1
> as well as GCC
> 4.8.1. The error is always the same.
>
> To learn more, I thought I'd take a look at the x264 port of OpenBSD 5.4,
> but found this
> in its Makefile, disabling all linking against both libav and ffmpeg as
> well as disabling
> all assembly (likely due to the binutils/gas issue):
>
>
> CONFIGURE_ARGS+=--disable-asm \
> --disable-ffms \
> --disable-gpac \
> --disable-lavf \
> --disable-swscale \
> --enable-static \
> --prefix=${PREFIX}
>
>
> The ffms thing disables linking against ffmpeg, the lavf+swscale stuff
> disables linking
> against libav. asm is self-explanatory.
>
> So the x264 port (as well as the precompiled package) are completely
> crippled. Not only is
> the assembly missing, costing tons of performance, but you can't even feed
> anything but
> raw video to it! What I need is the capability to feed stuff like
> H.264/AVC, MPEG2, VC-1
> videostreams etc. to x264, so I need libav or ffmpeg.
>
> Now, the main issue is the malloc() failure here. My home-brewn gas
> shouldn't be the
> problem, because it happens even when compiling from pure C/C++.
>
> My assumption would be, that maybe OpenBSDs libc implementation of
> malloc() behaves in
> ways that x264 can't handle properly?! I've tried looking at the x264
> source coude, but
> this stuff is just way beyond me, I don't understand any of the code
> really.
>
> I have so far managed to do this on NetBSD, FreeBSD/PC-BSD, MidnightBSD,
> Dragonfly BSD,
> OpenSolaris, Linux, Windows (CygWin and MinGW/MSYS), MacOS X and even
> Haiku OS with
> varying degrees of modifications to the build scripts and in one case a
> header file.
>
> It drives me crazy I can't figure this out on OpenBSD! ;) I've been trying
> this for
> months already!
>
> Does anybody have any idea on how I could proceed? I am no developer.. So
> yeah. If anybody
> would want to take a look at the actual source code, the latest x264
> version is available
> here:
>
>
> ftp://ftp.videolan.org/pub/x264/snapshots/last_x264.tar.bz2
>
>
> libav and ffmpeg can be obtained here (I prefer libav, but that's more a
> taste thing),
> one of them needs to be built first, as x264 needs to be linked against
> either of the two:
>
>
> http://git.libav.org/?p=libav.git;a=snapshot;h=HEAD;sf=tgz
> http://www.ffmpeg.org/releases/ffmpeg-2.2.tar.bz2
>
>
> The x264 trouble seems to originate in one of either "x264.c",
> "encoder/encoder.c" or
> maybe "common/common.c".
>
> Sorry for this very lengthy post, but I've tried and tried and tried and
> failed every
> time. When I finally got past the gas problem, I was sooo happy to get it
> built, only to
> hit this issue at runtime.
>
> I need help here, so if anyone has any idea on how to solve this, it'd be
> greatly
> appreciated!
>
> Not sure if it would make sense to contact the x264 port maintainer, a
Building libav/ffmpeg & x264 on 5.4
Greetings!
This is my first question here, please be gentle! ;)
So, as the subject line says, I am currently attempting to compile, install and
run
either the libav or ffmpeg media codec suite and then x264 on OpenBSD 5.4,
where x264
is then linked against either libav or ffmpeg. All of this on the x86_64/AMD64
architecture. x264 itself is a H.264/AVC video trans-/encoder.
So far so good.
The first big problem was the lack of a new enough GNU assembler ("as"/"gas"),
as x264
features SSSE3 inline assembly, that gas from binutils 2.15 cannot build. So I
went ahead
and compiled myself my own gas from binutils 2.24, which supposedly worked fine.
But the real bummer is what follows, and this error even shows up, if I disable
all
assembly optimizations in libav/ffmpeg as well as x264 (then it even compiles
on a stock
5.4 without my new gas 2.24, but won't run).
So, I start my video transcoding, and I get this (leaving out the [info] lines):
"x264 [error]: malloc of size 8856384 failed
x264 [error]: x264_encoder_encode failed
aborted at input frame 13, output frame 0"
I suspect x264 and not the libav/ffmpeg it was linked against, because 13 input
frames
were seemingly decoded by libav/ffmpeg, but not a single output frame was
encoded by x264.
Using older versions of x264/libav (up to 1 year old I tried) results in the
same problem,
only the malloc() size number is different. Like "31736" instead of the
"8856384" above.
Just in case you'd wanna know, this is a sample command line like the one I
called:
"x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f -2:0 --bitrate
1 --aq-mode 1 -p 1 --slow-firstpass --stats framestats.stats -t 2 --no-fast-pskip
--cqm flat input.264 -o pass1output.264"
I tried compiling this with and without assembly, and with both GCC 4.2.1 as
well as GCC
4.8.1. The error is always the same.
To learn more, I thought I'd take a look at the x264 port of OpenBSD 5.4, but
found this
in its Makefile, disabling all linking against both libav and ffmpeg as well as
disabling
all assembly (likely due to the binutils/gas issue):
CONFIGURE_ARGS+=--disable-asm \
--disable-ffms \
--disable-gpac \
--disable-lavf \
--disable-swscale \
--enable-static \
--prefix=${PREFIX}
The ffms thing disables linking against ffmpeg, the lavf+swscale stuff disables
linking
against libav. asm is self-explanatory.
So the x264 port (as well as the precompiled package) are completely crippled.
Not only is
the assembly missing, costing tons of performance, but you can't even feed
anything but
raw video to it! What I need is the capability to feed stuff like H.264/AVC,
MPEG2, VC-1
videostreams etc. to x264, so I need libav or ffmpeg.
Now, the main issue is the malloc() failure here. My home-brewn gas shouldn't
be the
problem, because it happens even when compiling from pure C/C++.
My assumption would be, that maybe OpenBSDs libc implementation of malloc()
behaves in
ways that x264 can't handle properly?! I've tried looking at the x264 source
coude, but
this stuff is just way beyond me, I don't understand any of the code really.
I have so far managed to do this on NetBSD, FreeBSD/PC-BSD, MidnightBSD,
Dragonfly BSD,
OpenSolaris, Linux, Windows (CygWin and MinGW/MSYS), MacOS X and even Haiku OS
with
varying degrees of modifications to the build scripts and in one case a header
file.
It drives me crazy I can't figure this out on OpenBSD! ;) I've been trying this
for
months already!
Does anybody have any idea on how I could proceed? I am no developer.. So yeah.
If anybody
would want to take a look at the actual source code, the latest x264 version is
available
here:
ftp://ftp.videolan.org/pub/x264/snapshots/last_x264.tar.bz2
libav and ffmpeg can be obtained here (I prefer libav, but that's more a taste
thing),
one of them needs to be built first, as x264 needs to be linked against either
of the two:
http://git.libav.org/?p=libav.git;a=snapshot;h=HEAD;sf=tgz
http://www.ffmpeg.org/releases/ffmpeg-2.2.tar.bz2
The x264 trouble seems to originate in one of either "x264.c",
"encoder/encoder.c" or
maybe "common/common.c".
Sorry for this very lengthy post, but I've tried and tried and tried and failed
every
time. When I finally got past the gas problem, I was sooo happy to get it
built, only to
hit this issue at runtime.
I need help here, so if anyone has any idea on how to solve this, it'd be
greatly
appreciated!
Not sure if it would make sense to contact the x264 port maintainer, as that
person seems
to have decided not to try and get it to work "properly" or maybe he hit the
same brick
wall I did and couldn't get past it? Is the issue maybe really not solvable at
all?
If so, I'd like to know and understand why at least.
Well, thanks a lot for any help you might be able to provide, and for reading
my wall of
text!
--
Michael Lackner
Lehrstuhl für Informationstechnologie (CiT)
Montanuniversität Leoben
Tel.: +43 (0)3842/402-1505 | Mail
