Picture Encodings

Introduction

Xpra supports a number of picture and video encodings provided you have the required libraries installed. Even then, the features of each encoding may vary based on the version of the libraries and other dependencies, both client and server side.
Here is the list as of v1.0

• lossless encodings:
  • rgb: rgb24 for regular windows, rgb32 for transparency support (compressed with zlib, lzo or lz4, just like packet compression)
  • png (24/32-bit colour), png/L (8-bit grayscale) and png/P (8-bit colour palette)
  • webp (lossless mode - deprecated)
• lossy encodings:
  • jpeg
  • webp (lossy mode - deprecated)
  • vpx (VP8 and VP9)
  • ​H264 (H264 is Magic): either using x264 or nvenc
  • h265 using x265 (not usable, far too slow)
  • xvid slower than x264 and vpx (see #1142 - deprecated)

Choosing an Encoding

The default mode is automatic and should be the best option in most cases: it will adapt to changes in both bandwidth conditions and content.

But if you have specific needs, the best thing to do is to try them all and choose the one that provides the best results.

Here are some rough guidelines:

• if you have the required hardware, use NVENC
• on LANs with 100Mbps or higher: rgb + zlib/lz4 should give you the best latency possible but consume quite a lot of bandwidth in the process..
• to save a little bit of bandwidth at the expense of framerate and latency but keep a lossless picture, try png - this is very CPU intensive
• otherwise, choose h264, vp8 or vp9 and tune the speed/quality to suit your needs (see below)

The other encodings are somewhat less useful:

• h265 via x265 is far too slow to encode
• jpeg gives lower size/quality than other lossy encodings, use this if the video encodings are not available and you need better compression than png or webp can give you

Related Links

This page is related to WindowRefresh, the Network and client rendering are also relevant for understanding the end to end pixel transmission process.

For video encodings, see vpx and h264. Good read: ​Falsehoods programmers believe about video

Before sending the pixels to the video encoder, xpra may need to first call a colourspace conversion step, (some newer versions of x264 support BGRA pixels as input directly) it is also required when downscaling the video. See: Colourspace conversion step - CSC

Tuning

The best way to choose the right option is through wiki/Testing.

Though you may find some ​good illustrations online to give you an idea of the trade offs. (ie: ​fps vs noise, ​fps vs size) Be aware that the lossless auto-refresh will trigger a lossless frame encoded using png or rgb. If the delay is too low, it may negate the benefits of using efficient compression. We have collected some encoding statistics with an older version.

Notes

When comparing performance, make sure that you use the right metrics... The number of updates per second is not always a good one (if there are more small regions, this can be a good or a bad thing), more examples here: Misleading Statistics

Future Encoding Support

We should eventually add support for:

• #617 f265 (h265)
• #454 daala - unlikely

Debugging

The first thing to check is the codec availability and version. On MS Windows, run the Encoding_Info.exe utility, or the bug report tool. On other platforms, run the xpra/codecs/loader.py script, and for debugging video the xpra/codecs/video_helper.py script. (both support a --verbose option).

When using non-video encodings, the encoding used for sending the pixels to the client will be the one which has been selected. The only exception to this rule is when the number of pixels is so small that trying to compress them would be pointless, and they are then usually sent as plain {{rgb}}} or png / webp.

With video encodings, things are more complicated: if there is a video region, the non video areas will use other encodings. Even the video region (which may be the whole window) will get automatically refreshed with a lossless encoding when it stops refreshing rapidly.

To debug encoding selection:

• the actual encoding used is logged with -d compress in the form:

  make_data_packet: image=XShmImageWrapper(BGRX: 149, 2, 6, 13), damage data: (1, 149, 2, 6, 13, 'rgb24')
  compress:   0.9ms for  499x316  pixels using  mmap with ratio   0.0% (  615KB to     0KB), delta=-1, client_options={'rgb_format': 'BGRX'}
  

• the current encoding set is best seen with: xpra info | grep encoding=
• to see statistics about which encodings are actually used: xpra info | egrep "last_used|total_frames|total_pixels"
• video region detection can be seen with xpra info | grep region