= Packet Encoding and Compression =

[[BR]]
See also:
* [/wiki/DataFlow Data Flow]: an overview of the data that flows over the network connection
* [/wiki/Network Network] for the network connection options
* [/wiki/NetworkProtocol Network Protocol] for the actual contents of the packets

[[BR]]

{{{#!div class="box"
== Wire Format ==
Each message may be split into multiple packets: large binary chunks (ie: compressed pixel data, large clipboard data, etc) will be sent in their own packet, skipping the generic compression layer.

[[BR]]
Each chunk is preceded by an 8 byte header containing:
* the magic value "P" (char value 80 in decimal, {{{0x50}}} in hexadecimal)
* one byte for protocol flags (stream encoder used, encryption, etc)
* one byte for compression level hint (0 for uncompressed)
* one byte for the chunk index (0 for main chunk)
* one long (4 bytes) for the data size that follows

[[BR]]
The main chunk containing the message uses index 0 and it is encoded (see bencode / rencode below).
The main message consists of a list of values, the first item is the packet type (see [/wiki/NetworkProtocol]).
The other chunks replace the item found at the specified index in the main chunk. (which should be empty)
[[BR]]

}}}

{{{#!div class="box"
== Bencode vs Rencode ==
The main chunk is encoded using one of those stream encoders:
* The old [http://xpra.org/trac/browser/xpra/trunk/src/xpra/net/bencode.py bencode] mode (see [http://en.wikipedia.org/wiki/Bencode Bencode on wikipedia]): a modified bencoder which supports non string keys in dictionaries, non ordered keys, and support for unicode
* The newer/faster [http://xpra.org/trac/browser/xpra/trunk/src/xpra/net/rencode rencode] mode, based on [https://code.google.com/p/rencode/ this version] with minor tweaks/fixes
* YAML (see r6934) which is much slower, but more widely available
[[BR]]
This allows various languages to implement the xpra protocol, specifying which encoder they want to use (bencode is more widely available than rencode).
We also include a faster bencoder implemented in Cython: [http://xpra.org/stats/bencode/ Cython vs Python bencoder]
}}}


{{{#!div class="box"
== Pixels and Icons ==

'''Note''': all window pixels, icons and cursors are sent in their own chunk using dedicated [/wiki/Encodings picture encodings] and bypass the stream encoder completely (for efficiency). Pixel compression is done in its own thread, to improve responsiveness.

The only exception to this rule is the `RGB` encoding, which will use the stream compression on the raw pixels (it still called from the pixel compression thread).
}}}


{{{#!div class="box"
== Compression ==
The compression level can be specified via the command line ("{{{-z LEVEL}}}" or "{{{--compress=LEVEL}}}").

Here is what the {{{LEVEL}}} does:
* {{{0}}} means no compression at all and is useful when bandwidth is no issue but latency is
* {{{1}}} is the best trade-off and will compress everything but without making too much of an effort
* values of 2 and above increase the compression (CPU usage and latency will increase) and should rarely be needed - the [/wiki/Encodings picture encoding] used is much more important
}}}


{{{#!div class="box"
== lz4 vs zlib ==
As of version {{{0.11.0}}}, xpra supports [https://pypi.python.org/pypi/lz4 lz4] compression and will use it instead of the default [http://docs.python.org/2/library/zlib.html zlib] when {{{LEVEL}}} is set to 1 and both ends support the compression mode. More information here: #443.
There is also support for [http://www.oberhumer.com/opensource/lzo/ lzo] in version {{{0.14.0}}} onwards, it can be used as an alternative to lz4: it is also much faster than zlib but not as efficient as lz4.
[[BR]]
There is a [/browser/xpra/trunk/src/tests/xpra/net/lz4_vs_lzo_vs_zlib.py benchmark test] you can run to measure the performance of lz4 and lzo compared to zlib. On average, lz4 is about 10 times faster than zlib on its fastest setting for regular packets, and even faster for RGB data (up to 60 times faster than zlib!) whilst also compressing better than zlib!
[[BR]]
You can also find some more generic statistics here: [http://xpra.org/stats/lz4/ lz4 vs zlib graphs]
}}}

{{{#!div class="box"
== python-lz4 / python-lzo Installation ==
We have an [/browser/xpra/trunk/rpmbuild/python-lz4.spec python-lz4 rpm specfile] and {{{RPM}}}s should be available for {{{Fedora}}} and {{{CentOS}}} 6.x in the RPM repository.
[[BR]]
Other distributions can install from source: [https://pypi.python.org/pypi/lz4 lz4].
Some distributions provide packages for {{{python-lzo}}}
[[BR]]
The MS Windows and Mac OSX binary installers include lz4 (all builds) and lzo (Python 2.x only).
}}}

{{{#!div class="box"
== Configuration ==

The default settings should end up using {{{lz4}}} and {{{rencode}}}, since those are by far the best options. That's why there are no command line options to change the defaults. The {{{zlib}}} and {{{bencode}}} options will remain for compatibility with clients which do not have support for them (ie: {{{Android}}}, html5..).

However, you can disable individual encoders/compressors using environment variables:
{{{
XPRA_USE_LZ4=0 xpra ...
XPRA_USE_BENCODER=0 xpra ...
XPRA_USE_RENCODER=0 xpra ...
}}}
With v0.11 onwards, you can also change the current encoder/compressor at runtime:
* switch to zlib: {{{xpra control :DISPLAY compression zlib}}}
* switch to lz4: {{{xpra control :DISPLAY compression lz4}}}
* switch to bencode: {{{xpra control :DISPLAY encoder bencode}}}
* switch to rencode: {{{xpra control :DISPLAY encoder rencode}}}
More info on {{{xpra control}}} here: #461.
You can view the current connection options with:
{{{
xpra info | egrep "connection.encoder=|connection.compression="
}}}
}}}