Xpra: Ticket #504: delegated encoding mode
Building on #370 and #426: we want to be able to do the encoding on the host (proxy server) using NVENC and let the actual xpra server sitting behind it just throw RGB pixels at it.
How this is going to work:
- the proxy server tells the real xpra server it will handle video encoding via a new capability attribute ("
encoding.proxy_video_encoder = { ... }
"?):
- how many encoding contexts it will accept
- what size requirements it has (min/max sizes, mask for non-even sizes, etc..)
- the real server then creates a "proxy video encoder" spec, which functions just like a real one, except that its characteristics are populated from the values above
- when this new encoder is requested to encode frames, it will just call the regular RGB encoder and add a tag to the metadata ("
proxy_video
"?) We will want the ability to choose whether to use compression or not from the real server to the proxy (virtual machines running on the same host as the proxy will probably not want the overhead, but servers connected over ethernet will)
- when the proxy receives such
draw
packets, it will create a video encoding context and compress the frame
- the proxy needs to watch for
lost-window
packets, and close the contexts if necessary
Potential problems:
- when running multiple virtual machines on the proxy server: large guest to host memory transfers will become a problem:
virtio-net
is good, but not that good. A 1080p screen at 25fps sent as RGBX will cost 200MB/s (and we do have to copy it more than once!), even a powerful server can't take too many of those
- creating an encoding context takes time: either we allow
draw
packets to be sent out of sync (and verify the window hasn't gone already by the time our packet is ready to send - what about resizes and moves?...), or we will suffer latency spikes. alternatively, we could create the context in advance, but we don't support context resizing yet... see #466
- latency perceived by the server: when the server receives the draw ACK packets, it will look like the client is taking a long time to process draw packets compared to other packets which go straight through the proxy - hopefully this won't cause us problems
- batch delay calculations: if we manage to send far more RGB frames to the proxy than it is capable of encoding and sending, the lack of ACK packets should smooth things out. I think.
- out of sync issues: just because we tell the real server how many video contexts it is allowed to create does not mean it will always honour it (for whatever reason), or that we won't have failures to create contexts proxy side. We could overcommit contexts too. So, we probably need to fallback to software encoding for these, hopefully rare, cases. I wouldn't want to have to add a message back to the real server to do realtime context tuning... but we may have to.
Longer term:
- load balancing: if the proxy server has a pair of K1 cards in it, we want to load balance
- overcommit and load distribution: a session that only does 2 fps should not monopolize an encoding context, we could give the encoding context to a more demanding session
Mon, 27 Jan 2014 14:15:22 GMT - Antoine Martin: attachment set
- attachment
set to proxy-encode-v1.patch
PoC proxy encoding delegation preparatory work
Mon, 27 Jan 2014 15:27:03 GMT - Antoine Martin: attachment set
- attachment
set to proxy-encode-v2.patch
PoC proxy encoding delegation: sends BGRX pixels with video proxy tag
Tue, 28 Jan 2014 06:59:35 GMT - Antoine Martin:
- r5286 (see detailed commit message) adds basic support for proxy encoding, tested with
x264
.
- r5275 ensures we can send RGB pixels uncompressed when setting speed to 100%, no matter how many pixels there are.
- r5276 and r5285 increase the network performance when dealing with large packets (which is the case for uncompressed RGB)
- r5287 + r5288 tidy things up
What still needs to be done:
- dealing with an encoding thread to reduce latency
- dealing with speed and quality changes - which are normally handled out-of-band...
- limit the number of contexts
Wed, 29 Jan 2014 16:02:40 GMT - Antoine Martin:
- fixes in r5300, r5307
- r5308 honours the quality and speed settings (with
x264
encoding)
- r5348 adds an encoding thread for better latency
- r5375 adds a proxy unix domain socket so we can get "
xpra info
" from the proxy process
- r5478 better logging
- r5483 lots of improvements and fixes (see commit message)
- the load balancing and NVENC issues are now here: #520
It is working surprisingly well!
Tue, 18 Feb 2014 09:52:38 GMT - Antoine Martin: owner changed
- owner
changed from Antoine Martin to Smo
Ready for testing (with and without #520):
- how many users can we run using this setup?
- what is the first bottleneck? (profile it?)
Tue, 10 Jun 2014 05:18:57 GMT - Smo: status changed; resolution set
- status
changed from new to closed
- resolution
set to worksforme
Closing this for now until we do some further testing. I'll reopen and comment when I have time to do more.
Sat, 23 Jan 2021 04:57:37 GMT - migration script:
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/504