Box86 performances

Like for last year, I will present a series of benchmark to try to show the efficiency (or inefficiency) of both box’s dynarecs. This year, I will benchmark both box86 and box64, to show how box86 evolved over 1 year of optimisations, and also show how box64, the new 64bits counterpart of box86 runs.

With box86 and box64, to check efficiency, I used 4 programs: 7z, dav1d, glmark2, and openarena.

• 7z is a widly used compression program, available on every distribution, and it comes with an integrated benchmark. The benchmark doesn’t seem to use much SSE code, and is very CPU performances centric. So it can be used to evaluate raw CPU power, and is considered a “worst case scenario” for both box (mostly CPU, no SSE, very little library function calls).
• dav1d is a video transcoding tools. It use highly optimized SSSE3 or NEON routine. This is the “nightmare scenario”: with hand-optimized assembly routine and mostly no wrapped function calls.
• glmark2 on the other hand is used to measure OpenGL performances. Here, CPU is not much used, and it’s mostly OpenGL calls all along. This is the ideal case scenario for box (mostly native library function calls).
• openarena is an open source game, based on idTech3 engine. It include a benchmark mode that will be used here. The point of this scenario is to show some “real case usage” of box.

7z benchmark

Method

The 7z used here is the version 16.02 found in the distribution. The x86 one comes from Debian, and the native one from the distribution (with the exeption of the Pandora, which was custom built). I noticed that 32bits ARM version of 7z have higher Decompression result than 64bits ARM64 version, while compression result are lower (I suspect asm optimisation maybe?).

The command 7z b was used, the output looks like that:

7-Zip [32] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,32 bits,4 CPUs LE)

LE
CPU Freq:   792  1486  1498  1495  1498  1492  1496  1493  1496

RAM size:    3776 MB,  # CPU hardware threads:   4
RAM usage:    882 MB,  # Benchmark threads:      4

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       3639   333   1062   3540  |      92329   396   1989   7877
23:       3539   338   1068   3606  |      90082   394   1976   7794
24:       3585   359   1075   3855  |      87605   395   1945   7691
25:       3506   367   1090   4004  |      84190   394   1901   7493
----------------------------------  | ------------------------------
Avr:             349   1074   3751  |              395   1953   7714
Tot:             372   1513   5733

I’ll report only the last number (so here 5733), and compare, per hardware, between native and emulated program.

box86 results

(1) limiting bench to 4 cores only.
(2) native was aarch64 in 2021.
(3) I suspect x86 test triggers some thermal throttle.
(4) I used a remote control in 2021. I have the physical machine in 2022.

box64 results

Conclusion

So, nothing spectacular on the 7zip case. Emulation speed is more or less the same. In fact, most of the optimisation have been around x87 and SSE code. Some optimisation around conditional jump is still needed (it was barely started), and it shows on this cpu-only benchmark. And the picture is the same for 32bits or 64bits emulation.

dav1d benchmark

Method

dav1d is a transcoding application, heavily optimized for SIMD and make use of SSSE on x86 (or SSE4 if availble) and NEON on ARM. While MMX, SSE and SSE2 convert fairly well to NEON, it’s not the case with SSSE3, conversion gets more complex resulting in emiting 4 or more NEON opcode per SSSE3 one (like pmaddubsw on XMM regs generate 10 NEON opcodes). Also, because the functions of dav1d are hand optimized assembler, the comparison between x86 and native will give lower number, as the dynarec convert 1:1 opcode, without reordering or removing any opcode. The resuling SSE->NEON code will be less efficiant than hand optimized routines. The benchmark will use the command ./dav1d -i Chimera-AV1-8bit-480x270-552kbps.ivf --muxer null, and just the resulting FPS will be taken.
When testing multiple thread, add --framethreads x --tilethreads x to the command, where the 2 x are the number of threads. The x86 and x86_64 version comes from a Debian distribution.

Example of result:

dav1d 0.7.1-91-gba875b9 - by VideoLAN
Decoded 8929/8929 frames (100.0%) - 37.11/1784.02 fps (0.02x)

You get ‘37.11’.

box86 results

(1) native build of dav1d is a custom build, with probably more optimisation than in regular version from distribution.
(2) native is aarch64 here.

box64 results

glmark2 benchmark

Method

glmark2 is a program to test OpenGL speed. A version targetting GLES2 hardware also exists, but because GLES2 is not supported by box86, only the Desktop Opengl version is used here. Some distribution (like debian) doesn’t include glmark2, so latest version from git source is used here (so 2020.04 version). For platform that doesn’t have full opengl driver, gl4es will be used. Note that glmark2 will crash when run on OpenGL 2.1 only (because it uses glGenerateMipmap function, but only get that function on OpenGL 3.0+), so MESA_GL_VERSION_OVERRIDE=3.2 is used on mesa driver that only expose 2.1 profile, or LIBGL_GL=30 when using gl4es.

The glmark2 score is simply used here, at default windowed 800x600 resolution (cropped to 800x480 on pandora). Because Mesa as evolved since last year, I will start fresh and not compared with 2021 result.

box86 results

(1) VSync cannot be disabled, lowering results.

box64 results

(1) VSync cannot be disabled, lowering results.

Conclusion

When running an app that is mainly relying on external libraries (that are wrapped), near native speeds can be acheived, with all result here being between 95% and 100% of native speed.

OpenArena benchmark

This test wasn’t present last year. But I did write an article about it there.

The benchmark uses some heavy graphics settings, making it quite GPU-limited on low-end machines, so I used an alternative graphic settings there (no bloom, refelction or shadows), to avoid beeing too GPU-limited, as that would invalidate the bench.

box86 results

(1) using a simplified graphic settings (no bloom, reflection or shadows), and fullscreen on device resolution
(2) at 2560x1440 resolution

box64 results

(1) at 2560x1440 resolution

Conclusion

On a game scenario, when you have reasonably optimized code and a good amount of function call that can be wrapped, the performances of both box are between 75% and 85% of the native version, which is satisfying!

Box86 speed

So in real world applications and games, things will be in between the two extreme test cases (7z and dav1d).

Note that things like SSE opcodes are converted to NEON ones, and most of the time a 1:1 conversion is done, making optimized SSE/SSE2 codes quite optimal in box86. It’s more difficult with SSE3/SSSE3, and converted code gets slower.
Wrapped functions gets native speeds, and emulated cpu will be at least 50% of what it could be if it was a native app (or even lower if the app/games contains ASM hand optimized routines). The resulting speed will be something in between depending of the percentage of emulated code vs wrapped lib ratio…