x264 benchmark compilation & installation guide for NetBSD >=6.0 RC2
There is one thing that should be mentioned beforehand and that is that the guide presented here has NO validity for OpenBSD <=5.1 or FreeBSD <9.1 or any older version. As of now, restrictions of the GNU portable assembler on those systems have prevented me from reaching a satisfactory result on OpenBSD and FreeBSD. So for now, of all BSD style systems, the compilation and installation of libav and x264 only works on NetBSD >=6.0 and also DragonFly BSD >=3.0.
To compile and install the x264 benchmark on NetBSD, some basic software needs to be present first. To install what is required, we will use the BSD package manager, which will need to be configured first. At the time of writing, the benchmark required NetBSD 6.0 release candidate 2, since the final version 6.0 wasn't released yet. So we will assume version 6.0 RC2 here, the path we're about to set will need some adjustment for newer versions of course. In our hypothetical case we will also assume the use of a Core 2 Duo processor, so we will choose the x86_64 architecture. If you have a different CPU, you might need to adjust that according to your exact platform. To do the following package installation, you need to be the superuser root:
- export PKG_PATH="ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/6.0_RC2_2012Q3/All/"
- pkg_add autoconf
- pkg_add gmake
- pkg_add yasm
- pkg_add wget
- pkg_add git
- pkg_add easygit
- pkg_add bzip2
- pkg_add firefox (adds some useful depedency libraries)
- pkg_add bash
- pkg_add pkg-config
- pkg_add texi2html
Since the configuration scripts of the software we're about to build are not extremely well prepared for NetBSD, we will go the easy way, and just create the default installation directory ourselves and then add it to the sessions search path. As it is with PKG_PATH, you shouldn't close your active terminal session after setting the PATH variable. If you do, you will need to set those variables again when opening a new terminal session:
- mkdir /usr/local
- export PATH="$PATH:/usr/local/bin"
No let's go for the real thing:
The benchmarking software itself
To be able to use x264, we need the decoder filters from libavcodec (in short libav) and of course the x264 encoder itself, which will be linked against libav during the course of configuration and compilation. Additionally, we need the benchmark input video, download mirrors for that are available at the bottom of the x264 [benchmark results list]. If you like, you may also use ffmpeg/ffms as a replacement for libav.
For simplicities sake, we shall assume, that the files are now placed in /home/user/x264src/. So on the terminal we switch to that directory and start with libav. Please unpack the archive like tar -xzvf <filename.tgz>, after which you will see a new libav subfolder. Change to that new folder, so that you can begin to do the necessary modifications and configurations:
A Perl script, which is a part of the libav source code and which is responsible for generating the documentation for the code has a wrong shebang line in its code. The shebang line is the very first line of the script, that we will need to change. The script is in a subfolder of the libav folder called doc and itself is called texi2html.pl. We shall edit that script using vi or maybe xedit if you have a graphical user interface and change that first line, search for:
#! /usr/bin/perl -w
Let's just replace that with the path that the Perl binary can be found in on NetBSD:
#! /usr/pkg/bin/perl -w
After that simple modification we may configure, compile and install libav:
- ./configure --prefix=/usr/local --enable-gpl --enable-nonfree
- gmake install
Now we have got libav installed, and can thus continue with x264.
First, after changing back to /home/user/x264src/ let's unpack the x264 source code like: tar -xjvf <x264 archive>. Now change to the newly created x264 subfolder.
We will need to apply a dirty little trick for x264 that affects at least the configure script itself, but potentially a lot more in that source tree. Basically, all shebang lines pointing at the bash shell will be wrong, as the Bash shell is installed on a weird, different path on NetBSD. Instead of fixing them all, let's just create a link to the Bash in the directory that the scripts are expecting:
- ln -s /usr/pkg/bin/bash /bin/bash
After that, we should set some CFLAGS and CXXFLAGS for the C/C++ compiler. These are quite generic and should apply to most processor microarchitectures that NetBSD supports:
- export CFLAGS="-O3 -ffast-math -march=native -mtune=native -I/usr/local/include -L/usr/local/lib
- export CXXFLAGS="-O3 -ffast-math -march=native -mtune=native -I/usr/local/include -L/usr/local/lib
After thet, let's run the configuration:
- ./configure --prefix=/usr/local
After that step you should check, whether the output of the script shows the line "lavf yes". x264 wlll only link correctly against libav if that is the case. If it is indeed not the case, you will need to debug the linker, analyzing the output found in config.log. If it looks ok, we will still need to edit the newly created file config.mak, since the configure script will most likely have forgotten a small but important detail here. So please look for a line near the end of that file, that looks somewhat like this:
LDFLAGSCLI = -L. -lavformat -lavcodec -lswscale -lavutil -lm -lz -lbz2 -lpthread -lswscale -lavutil
We will need to add a little something right there:
LDFLAGSCLI = -L. -L/usr/local/lib -lavformat -lavcodec -lswscale -lavutil -lm -lz -lbz2 -lpthread -lswscale -lavutil
After that change, you may build and install x264:
With that done, x264 is installed and ready to take off! The functionality of the software can be checked by running x264 --version. If that works and it shows as being linked against libswscale and libavformat, then all went smoothly. Now please download the following benchmark script:
This file launchbenchmark.sh needs to be copied into the subfolder dedicated to the benchmark, if it does not yet exist, please just created it: mkdir -p /home/user/x264benchmark. You should also put the input video elephantsdream_source.264 into that folder.
Now unfortunately x264 will use a way of CPU core detection that might very well work fine on Linux and some flavors of Unix, but will fail on BSD and with it on NetBSD. That results in x264 having strictly no idea how many logical processors there are in the system.
When that happens, x264 will only launch a single worker thread, and hence will only load one logical processor core, even if we have multithreading support on board. Luckily, there is an easy fix for this, we just need to add a little something to our
launchbenchmark.sh, telling x264 how many worker threads it's supposed to spawn. For that, we should follow the old x264 rule:
- Number of worker threads = logical CPUs * 1.5
So, for a CPU with 4 cores, that would make 6 threads. For a CPU with 4 cores and hyper threading (=8 logical CPUs) that would make 12 threads. If by any chance you have a triple core that simple old formula won't work of course, but in that case just round up to 5 threads.
The maximum number of workers is by the way 32, at least at the time of writing, so you can't spawn more worker threads than that. So to make that little change to the benchmark, open up launchbenchmark.sh with vi or maybe xedit, and add the option after both occurences of
Here, n is the amount of worker threads, so e.g. --threads 12 for a system with 8 logical CPUs. Now with that, nothing should stand in the way of using multiple CPUs, cores and hyper threads anymore!
So now you can basically just launch the benchmark, so just enter the following line after ensuring that the script is executable (chmod +x ./launchbenchmark.sh) and while of course sitting in /home/user/x264benchmark:
- time -p ./launchbenchmark.sh
That could look somewhat like this (click to enlarge). The result can be read from the terminal after the benchmark is done, the important value here is the one that will be shown besides the string "Real":
Depending on the host machine that might take some hours, in extreme cases even days, weeks or months. If you have problems with this guide or if there are any additional questions about the x264 benchmark on NetBSD/BSD, please refer to the [according forum thread] and ask there. The forum is german, but if you ask in english, you will still get a reply, at least by me. :)
[Back to the x264 benchmark results list]
[Comment on this guide]