Xpra: Ticket #639: UDP transport

Rather than using mosh #219, we should be able to do our own UDP transport: we know which packets must arrive and may need to be re-sent (window metadata, etc), and which ones we can just skip when they go missing: we just send a newer update instead (window pixels, cursors, etc..).

Then we can also tune the video encoders and insert key frames when needed (and maybe also lower the default gap between key frames when UDP is used), etc..

This may also helps us achieve other things: we could more easily use parallel processes for encoding since each process can then send its data without needing to synchronize or share the socket. Also useful when using hardware encoders which have their own network ports.

We should not need to worry about UDP hole punching for now, if ever. I don't think we want to be using a library like UDT either: we would lose some flexibility, and the python bindings are old and not very portable...

Fri, 26 Dec 2014 04:28:50 GMT - Antoine Martin: status changed

Some preliminary notes:

very distant future: handle UDP broadcasts for multiple clients

Tue, 14 Apr 2015 16:18:08 GMT - Antoine Martin: milestone changed

Not as useful as first thought, too many implementation issues, other more pressing issues (ie: #835) would clash with this. So re-scheduling as future.

Wed, 17 Feb 2016 09:58:44 GMT - Antoine Martin:

For the encryption part, we can use CBC with a new IV based on the packet sequence number, or a counter mode, or just DTLS.

Wed, 17 Feb 2016 10:58:44 GMT - Antoine Martin:

See also #1124

The packet loss rate may have an effect on our encoding selection: maybe we should use encodings that tolerate packet loss better. ie: not png or jpeg for big areas. We already keep track of what hasn't been acked yet, and we can re-send if necessary.

Sun, 19 Feb 2017 06:43:13 GMT - Antoine Martin: milestone changed

Mon, 10 Jul 2017 14:14:48 GMT - Antoine Martin: milestone changed


Mon, 31 Jul 2017 09:12:50 GMT - Antoine Martin:

This has some helpful pointers: How to stream H.264 video over UDP using the NVidia NVEnc hardware encoder?

Thu, 17 Aug 2017 09:58:32 GMT - Antoine Martin:

In order to do DTLS, we would need to use something like issue 3501 - circa 2.0?) - problem is that >=1.9 doesn't build on macos: #1544, and is a much lower level API.

Thu, 17 Aug 2017 13:55:27 GMT - Antoine Martin: attachment set

udp and dtls work in progress: client can send hello wrapped in a UDP frame

Sat, 19 Aug 2017 09:00:55 GMT - Antoine Martin: attachment set

working udp connections (but without handling client authentication, packet loss, etc..)

Sun, 20 Aug 2017 10:13:08 GMT - Antoine Martin: attachment set

work in progress: hooks for handling packet failures, mtu detection

Mon, 21 Aug 2017 15:39:43 GMT - Antoine Martin: attachment set

kinda working path - with many limitations

Mon, 21 Aug 2017 16:00:15 GMT - Antoine Martin:

Here's how the UDP patch above works:

Still TODO:

Good read: Dealing with IPv6 fragmentation in the DNS: UDP is different, and in UDP a functional response to path message size issue inevitably relies on interaction with the upper-level application protocol.

Tue, 22 Aug 2017 12:31:36 GMT - Antoine Martin: attachment set

now with timer scheduling and decent performance

Wed, 23 Aug 2017 08:04:17 GMT - Antoine Martin: attachment set

add some rfb refactoring

Fri, 25 Aug 2017 17:42:07 GMT - Antoine Martin: attachment set

add some authentication refactoring

Sat, 26 Aug 2017 06:13:26 GMT - Antoine Martin: attachment set

more authentication refactoring: add d3des as a digest option ("des")

Sat, 26 Aug 2017 15:31:49 GMT - Antoine Martin: attachment set

more asynchronous packets, rfb authentication integration, disabled delta, trim protocol classes, fix pydev warnings, etc.

Sun, 27 Aug 2017 04:46:57 GMT - Antoine Martin:

Merged the rfb parts and some preparatory work in r16710 + r16712. (see ticket:1620#comment:2)

Sun, 27 Aug 2017 05:44:36 GMT - Antoine Martin: attachment set

updated patch for r16713

Sun, 27 Aug 2017 12:59:52 GMT - Antoine Martin: attachment set

fixed initial packet errors (asynchronous handling enabled too early), fix subprocess wrapper code

Sun, 27 Aug 2017 13:15:01 GMT - Antoine Martin:

With the patch above, UDP seems to work quite reliably, even with a high packet loss (10%). Tested with:

xpra start-desktop --start=xterm --bind-udp= -d udp
XPRA_UDP_DROP_PCT=10 xpra attach udp:localhost:10000 --speaker=no -d udp --remote-logging=no

(adding XPRA_USE_ALIASES=0 is useful for debugging network traffic with debug logging or with tcpdump)

Things left to fix:

Tue, 29 Aug 2017 10:30:16 GMT - Antoine Martin: attachment set

more aggressive control packet scheduling, calculate which chunks should have arrived, option to test missing the first packet - which requires more tricky udp-control packet scheduling

Tue, 29 Aug 2017 14:40:26 GMT - Antoine Martin:

Code committed in r16734 - see large commit message.

This code also adds a new flag:


(set a higher value to drop the first packet more than once) This was very useful for testing the early asynchronous re-send logic and uncovered a few subtle bugs. Performance is still pretty decent, even with 10% packet loss at both ends, and without doing any profiling or optimization! The code is quite tricky to understand, so there are a lot more docstrings than usual in there. Basic information has also been added to wiki/Network.

Still TODO (probably not all for this release):

Thu, 28 Sep 2017 04:46:27 GMT - Antoine Martin:

Minor enhancements in r16969.

For audio, it may be possible to just let gstreamer handle the dropped and out-of-order packets via AAC/RTP streaming:

gst-launch-1.0 audiotestsrc ! audioconvert ! avenc_aac ! rtpmp4apay ! udpsink
gst-launch-1.0 udpsrc ! application/x-rtp,clock-rate=44100,config=40002410adca00 ! \
    rtpjitterbuffer ! rtpmp4adepay ! avdec_aac ! audioconvert ! autoaudiosink

Having to specify the caps on the receiver is a pain though.

Thu, 05 Oct 2017 15:09:14 GMT - Antoine Martin:

For DTLS, wolfcrypt-py might be useful.

Tue, 24 Oct 2017 16:04:45 GMT - Antoine Martin: owner, status changed

This will have to do for this release, will follow up in #1669.

@maxmylyn: can you break it? (bear in mind that this isn't meant to compete with TCP at this point - just to be usable)

Fri, 27 Oct 2017 21:57:43 GMT - J. Max Mena: status changed; resolution set

Running a trunk r17263 Fedora 25 server and client:

Unable to break it, and performance feels pretty good, at least almost as good as TCP.

Since there's another ticket for following up, I'm going to go ahead and close this one.

Sun, 22 Jul 2018 08:54:48 GMT - Antoine Martin:

New wiki page: UDP.

Sat, 23 Jan 2021 05:01:38 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/639