xpra icon
Bug tracker and wiki

Opened 3 years ago

Last modified 3 weeks ago

#639 assigned task

UDP transport

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: major Milestone: 2.2
Component: core Version: trunk
Keywords: Cc:

Description

Rather than using mosh #219, we should be able to do our own UDP transport: we know which packets must arrive and may need to be re-sent (window metadata, etc), and which ones we can just skip when they go missing: we just send a newer update instead (window pixels, cursors, etc..).

Then we can also tune the video encoders and insert key frames when needed (and maybe also lower the default gap between key frames when UDP is used), etc..

This may also helps us achieve other things: we could more easily use parallel processes for encoding since each process can then send its data without needing to synchronize or share the socket. Also useful when using hardware encoders which have their own network ports.

We should not need to worry about UDP hole punching for now, if ever.
I don't think we want to be using a library like UDT either: we would lose some flexibility, and the python bindings are old and not very portable...

Attachments (12)

udp.patch (31.1 KB) - added by Antoine Martin 5 weeks ago.
udp and dtls work in progress: client can send hello wrapped in a UDP frame
udp-v2.patch (35.0 KB) - added by Antoine Martin 4 weeks ago.
working udp connections (but without handling client authentication, packet loss, etc..)
udp-v3.patch (49.0 KB) - added by Antoine Martin 4 weeks ago.
work in progress: hooks for handling packet failures, mtu detection
udp-v4.patch (63.3 KB) - added by Antoine Martin 4 weeks ago.
kinda working path - with many limitations
udp-v5.patch (67.9 KB) - added by Antoine Martin 4 weeks ago.
now with timer scheduling and decent performance
udp-v6.patch (89.4 KB) - added by Antoine Martin 4 weeks ago.
add some rfb refactoring
udp-v7.patch (145.7 KB) - added by Antoine Martin 4 weeks ago.
add some authentication refactoring
udp-v8.patch (162.7 KB) - added by Antoine Martin 3 weeks ago.
more authentication refactoring: add d3des as a digest option ("des")
udp-v9.patch (194.3 KB) - added by Antoine Martin 3 weeks ago.
more asynchronous packets, rfb authentication integration, disabled delta, trim protocol classes, fix pydev warnings, etc.
udp-v10.patch (61.1 KB) - added by Antoine Martin 3 weeks ago.
updated patch for r16713
udp-v11.patch (62.0 KB) - added by Antoine Martin 3 weeks ago.
fixed initial packet errors (asynchronous handling enabled too early), fix subprocess wrapper code
udp-v12.patch (66.7 KB) - added by Antoine Martin 3 weeks ago.
more aggressive control packet scheduling, calculate which chunks should have arrived, option to test missing the first packet - which requires more tricky udp-control packet scheduling

Download all attachments as: .zip

Change History (24)

comment:1 Changed 3 years ago by Antoine Martin

Status: newassigned

Some preliminary notes:

  • we can't do encryption to begin with (see #198, #584) because we use AES CBC: In CBC mode, each block of plaintext is XORed with the previous ciphertext block before being encrypted. This way, each ciphertext block depends on all plaintext blocks processed up to that point. (this breaks as soon as we get packets out of order, or if we drop any..)
  • if we don't get an ACK from the client after a delay, we must probe it (we could augment the paint packet with a new capability)
  • pass flag to video encoders so they can enable support from dropped frames?
  • maybe use a new command line argument: --bind= which would bind to both TCP and UDP ports?
  • how do we take into account the maximum UDP packet size (and detect or change it..)
  • the protocol class would need new threads for UDP: read and write (assuming each end does both - not necessarily the case..)
  • client will need to wait for the paint packets to arrive in the right order, and timeout if one goes MIA, deal with dupes - we could also decode them (at least some of them... like non video) whilst waiting
  • client can ask server for a refresh when things go wrong, we already reset the video pipeline when the client reports decoding errors

very distant future: handle UDP broadcasts for multiple clients

comment:2 Changed 2 years ago by Antoine Martin

Milestone: 0.15future

Not as useful as first thought, too many implementation issues, other more pressing issues (ie: #835) would clash with this. So re-scheduling as future.

comment:3 Changed 19 months ago by Antoine Martin

For the encryption part, we can use CBC with a new IV based on the packet sequence number, or a counter mode, or just DTLS.

comment:4 Changed 19 months ago by Antoine Martin

See also #1124

The packet loss rate may have an effect on our encoding selection: maybe we should use encodings that tolerate packet loss better. ie: not png or jpeg for big areas.
We already keep track of what hasn't been acked yet, and we can re-send if necessary.

comment:5 Changed 7 months ago by Antoine Martin

Milestone: future2.1

comment:6 Changed 2 months ago by Antoine Martin

Milestone: 2.12.2

re-scheduling

comment:8 Changed 5 weeks ago by Antoine Martin

In order to do DTLS, we would need to use something like PyDTLS (no API docs, no examples, no commits for 4 months, no python3 support) or use the raw openssl via python-cryptography (support added in issue 3501 - circa 2.0?) - problem is that >=1.9 doesn't build on macos: #1544, and is a much lower level API.

Last edited 5 weeks ago by Antoine Martin (previous) (diff)

Changed 5 weeks ago by Antoine Martin

Attachment: udp.patch added

udp and dtls work in progress: client can send hello wrapped in a UDP frame

Changed 4 weeks ago by Antoine Martin

Attachment: udp-v2.patch added

working udp connections (but without handling client authentication, packet loss, etc..)

Changed 4 weeks ago by Antoine Martin

Attachment: udp-v3.patch added

work in progress: hooks for handling packet failures, mtu detection

Changed 4 weeks ago by Antoine Martin

Attachment: udp-v4.patch added

kinda working path - with many limitations

comment:9 Changed 4 weeks ago by Antoine Martin

Here's how the UDP patch above works:

  • we aggregate packet chunks into one buffer (and so we lose zero copy...)
  • we send this buffer to the other end via UDP, after splitting it into chunks that fit into the MTU (which we can get client-side since we call connect() and we send that value to the server via udp-control packets)
  • when receiving, we reconstruct the original buffer - caching chunks until we have all of them
  • if we are missing some packets (missing sequence number) or chunks, we send a resend request
  • some packet types (in particular the UDP control packet) don't need to honour the sequence number
  • other packets are not resent, we just tell the other end to forget them and we send a new one instead (with a new sequence number)

Still TODO:

  • (re)schedule udp-control packets based on observed latency (currently every set at 2 seconds for debugging)
  • uuid and protocols: should the client make an initial request for one? (syncookie like?)
  • drop AES? (it's a pain and untested)
  • drop DTLS? (or fix it - behaves more like a stream?)
  • drop packet accounting hooks (not reliable anyway?)
  • mmap cannot be used (needs reliable delivery to clear after use)
  • refactor the protocol class and think about RFB: #1620
  • cythonize the protocol classes?
  • drop protocol format / parse threads? (we can compress the clipboard in the encode thread)
  • threading races with packet structures?
  • handle client roaming: update target address on server (and reset statistics?)
  • test over wifi, DSL, 3G, etc
  • turn off delta (may fail if we missed the previous paint it relies on), expose "asynchronous" capability of protocol?
  • verify which audio codecs deal with missing packets gracefully, disable the others when using UDP
  • more asynchronous packets (from wiki/NetworkProtocol): bell, notify close, info-response, close-window, key-action, key-repeat, server-settings, info-request, encoding, etc.
  • add more packet fail callbacks. Resend: cursors, window icons, pointer position, ping. Drop: audio data
  • maybe add logic to choose between sending a new draw packet and sending just the missing chunks (cheaper)?
  • inband udp control via udp header bitmap, so receiver knows a packet was optional / asynchronous without needing / waiting for a "udp-control" packet?
  • paint ack to be merged with udp-control?
  • ping and ping-echo merged too? piggyback?
  • "udp-control" should not need to be a "ui-packet" server-side - could cause race conditions
  • add bitmap of preceding packets so udp receiver knows which packets it doesn't need to wait for (those we can resend anyway)
  • queued priorities? (clipboard packets only need to wait for each other, not paint, etc)
  • video: try decoding partial packets? only if we have the header? wait only a little bit for those?
  • prevent resends from accumulating and causing the "udp-control" packets to grow too big

Good read: Dealing with IPv6 fragmentation in the DNS: UDP is different, and in UDP a functional response to path message size issue inevitably relies on interaction with the upper-level application protocol.

Last edited 3 weeks ago by Antoine Martin (previous) (diff)

Changed 4 weeks ago by Antoine Martin

Attachment: udp-v5.patch added

now with timer scheduling and decent performance

Changed 4 weeks ago by Antoine Martin

Attachment: udp-v6.patch added

add some rfb refactoring

Changed 4 weeks ago by Antoine Martin

Attachment: udp-v7.patch added

add some authentication refactoring

Changed 3 weeks ago by Antoine Martin

Attachment: udp-v8.patch added

more authentication refactoring: add d3des as a digest option ("des")

Changed 3 weeks ago by Antoine Martin

Attachment: udp-v9.patch added

more asynchronous packets, rfb authentication integration, disabled delta, trim protocol classes, fix pydev warnings, etc.

comment:10 Changed 3 weeks ago by Antoine Martin

Merged the rfb parts and some preparatory work in r16710 + r16712. (see ticket:1620#comment:2)

Changed 3 weeks ago by Antoine Martin

Attachment: udp-v10.patch added

updated patch for r16713

Changed 3 weeks ago by Antoine Martin

Attachment: udp-v11.patch added

fixed initial packet errors (asynchronous handling enabled too early), fix subprocess wrapper code

comment:11 Changed 3 weeks ago by Antoine Martin

With the patch above, UDP seems to work quite reliably, even with a high packet loss (10%).
Tested with:

xpra start-desktop --start=xterm --bind-udp=0.0.0.0:10000 -d udp
XPRA_UDP_DROP_PCT=10 xpra attach udp:localhost:10000 --speaker=no -d udp --remote-logging=no

(adding XPRA_USE_ALIASES=0 is useful for debugging network traffic with debug logging or with tcpdump)

Things left to fix:

  • connection fails if the packet loss happens early and server side
  • test over wifi, DSL, 3G, etc
  • performance: profile it, cythonize it, etc
  • audio codecs tests
  • add code to try to re-order audio packets (or generalize into a multiqueue?)
  • maybe merge some packet: ping, udp-control, paint acks
  • man page and wiki updates
Last edited 3 weeks ago by Antoine Martin (previous) (diff)

Changed 3 weeks ago by Antoine Martin

Attachment: udp-v12.patch added

more aggressive control packet scheduling, calculate which chunks should have arrived, option to test missing the first packet - which requires more tricky udp-control packet scheduling

comment:12 Changed 3 weeks ago by Antoine Martin

Code committed in r16734 - see large commit message.

This code also adds a new flag:

XPRA_UDP_DROP_FIRST=1 xpra ...

(set a higher value to drop the first packet more than once)
This was very useful for testing the early asynchronous re-send logic and uncovered a few subtle bugs.
Performance is still pretty decent, even with 10% packet loss at both ends, and without doing any profiling or optimization!
The code is quite tricky to understand, so there are a lot more docstrings than usual in there.
Basic information has also been added to wiki/Network.

Still TODO (probably not all for this release):

  • the jitter value is critical, it is currently hard-coded at 20ms - we will need to derive it somehow (hard without knowing which packets are re-sends... we could live patch the header?)
  • audio packets are not re-ordered, but they should be - we will need a small queue, and cap it using the jitter value
  • test over wifi, DSL, 3G, etc
  • performance: profile it, cythonize it, etc
  • audio codecs tests
  • better integration with pixel encoders: switch on error concealment (more key frames), this may require using a container format rather than plain H264... (similar to #1463 - using the ffmpeg encoder)
  • maybe merge some packet: ping, udp-control, paint acks - all a bit redundant!
Note: See TracTickets for help on using tickets.