x264 Encoder

The x264 encoder supports many options. H264 is the default encoding used when it is available because it offers the best performance and compression out of the box, unless you have NVENC.

Hopefully, this will one day be superseded by AV1 (#1668), until then, it is the best encoding you can use with xpra. (see also: vp9 #464)

Quality/speed and minimum quality/speed can be set via the command line or via the tray menu. Note that even when using the x264 encoding, some small screen updates may get sent as webp, png or even just plain RGB to save time/bandwidth (not avoid encoding a full video frame).

Recent versions of x264 support RGB pixels directly as input, which removes the need for a CSC step on the server, and the OpenGL rendering in the client also handles RGB data directly - which allows for an even faster forwarding of pixels to the client's screen. `Note: at lower quality settings, either automatically or manually selected, a CSC step may be introduced to reduce bandwidth by downsampling the input to the encoder.

Installation

When building xpra from source, x264 is included by default.

For encoding support (server), you only need the x264 library and headers installed.

For decoding support (client), you need the ​ffmpeg library and headers installed.

Unless you have also installed nvenc, x264 will be automatically used when you select the h264 or auto encodings. Alternatively, you can use the --video-encoders=x264 option to ensure that x264 is the only video encoder loaded in the server.

Links

• ​x264 homepage
• ​H.264 encoding guide for more information on x264 options
• ​wikipedia H.264

The following wiki pages are also relevant:

• wiki/WindowRefresh - when and how the screen updates are processed
• wiki/Encodings - all the picture encodings available
• wiki/CSC - the Colourspace conversion step

Profiles

(see ​wikipedia h264 profiles) x264 supports the following profiles: baseline, main, high, high10, high422 and high444.

At present, we use high or better by default unless the client overrides it.

Quality Option

What this does should be obvious (in x264 speak, this controls the rc.f_rf_constant parameter), but it is more complicated than you think:

• we support 3 different colourspace modes: YUV420, YUV422 and YUV444 and this affects the quality of the picture too. (YUV420 is used for lowest quality settings). YUV444 uses roughly twice as much bandwidth as YUV420. Switching from one mode to another is expensive, as we then need to send a new key frame and re-initialize both the encoder and the decoder - so the thresholds for going up to the next mode are not the same as the thresholds for going down to the next mode (prevents a yoyo effect).
• not all modes are supported by all profiles, so we need to switch to a different profile to support YUV422 (high422 or high444) and YUV444 (high444 only).
• some builds against older versions of libav/ffmpeg only support YUV420, see x264-limited-csc.patch

Speed Option

This option, shown as "latency" via the tray menu, controls how hard the encoder is going to work at compressing the picture. Working harder means lower bandwidth, but also higher latency. x264 supports the following speed settings: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow and placebo. xpra maps the 100% to 0% speed option from ultrafast to slower only. veryslow and placebo are not particularly useful (diminishing returns: much much slower and without sufficient savings for real-time use). ultrafast is only available when setting the encoder speed to 100% manually (via the command line or UI) - note that this setting has side-effects which prevents other settings from behaving as they should if the option is later changed.

Tickets

Support for x264, especially on the decoding side via avcodec, has provided an almost constant stream of bugs:

• #462 x264 encoder threading related memory corruption and crash
• #457 client memory leak
• #449 Crash when connecting with x264 encoding
• #427 draw decode failure when attaching to older remote
• #421 fix support for libav 9.8
• #398 AVFrameWrapper falling out of scope before being freed by avcodec

etc..

Ideas for future Work

• better tuning: most default options are targeted at video
• motion algorithm improvements:
  • tweak motion range (see ​merange)
  • use a better motion algorithm where appropriate: esa (exhaustive search algorithm) is expensive, dia (diamond search) is a better fit for traditional scrolling
  • write one where we can feed the information ourselves (assuming we do get it somehow)
• tune input-range (see ​X264 Settings: input-range

etc..