Xpra: Ticket #1839: fix broadway software h264 decoding

Split from #1463, the error that comes up is RangeError: Source is too large.



Sun, 13 May 2018 05:45:49 GMT - Antoine Martin: status changed

The error RangeError: Source is too large turned out to be trivial to fix: r19294, just needed a "new" pair of eyes, found it when looking into mpeg1 decoding (#1816).

So r19295 enables broadway software decoding by default, again.

What we may still want to do here:


Sun, 13 May 2018 07:01:39 GMT - Antoine Martin: owner, status changed

The html5 client already sets the correct h264 encoding properties (see also #1840):

"encoding.h264.YUV420P.profile"		: "baseline",
"encoding.h264.YUV420P.level"		: "2.1",
"encoding.h264.cabac"			: false,
"encoding.h264.deblocking-filter"	: false,

And these are correctly applied by the x264 codec (as seen with "-d x264"):

x264 context=0x7f08788859c0, YUV420P  640x356  quality=40, speed=40, source=unknown
 preset=medium, profile=baseline, tune=zerolatency
 me=DIA, me_range=16, mv_range=-1, weighted-pred=0
 b-frames=0, max delayed frames=0
 vfr-input=False, lookahead=-1, sync-lookahead=0, mb-tree=False, bframe-adaptive=FAST
 open-gop=True, bluray-compat=False, cabac=False, deblocking-filter=False
 intra-refresh=False, interlaced=False, constrained_intra=False
 threads=auto, sliced-threads=True
x264 encode YUV420P frame     0 as  IDR slice with 8 nals, tune=zerolatency, total   10776 bytes, keyframe=True , delayed=0

Note: nvenc doesn't have such controls, so it may or may not produce a stream which is more CPU intensive to decode.

So the only remaining issue is the blacklisting, which we can look at after some testing. This:

Decoder.js:450 Invalid asm.js: Unexpected token

could slow things down quite a bit - not sure how to fix that (building it from source is a pain).


Tue, 15 May 2018 10:09:42 GMT - Antoine Martin:

Updates:

Which gives us:

window.5.encoder=x264
window.5.encoder.b-frames=0
window.5.encoder.delayed=11
window.5.encoder.fast-decode=True
window.5.encoder.formats=('BGRX', 'YUV422P', 'YUV420P', 'BGRA', 'YUV444P')
window.5.encoder.fps=16
window.5.encoder.frame-types.IDR=1
window.5.encoder.frame-types.P=7
window.5.encoder.frames=8
window.5.encoder.generation=9
window.5.encoder.height=432
window.5.encoder.lossless=False
window.5.encoder.max-size=(8192, 4096)
window.5.encoder.ms_per_frame=1
window.5.encoder.params.bframe-adaptive=NONE
window.5.encoder.params.bluray-compat=False
window.5.encoder.params.cabac=False
window.5.encoder.params.constrained_intra=False
window.5.encoder.params.deblocking-filter=False
window.5.encoder.params.interlaced=False
window.5.encoder.params.intra-refresh=False
window.5.encoder.params.lookahead=0
window.5.encoder.params.mb-tree=False
window.5.encoder.params.me.me-range=16
window.5.encoder.params.me.mv-range=256
window.5.encoder.params.me.type=HEX
window.5.encoder.params.me.weighted-pred=0
window.5.encoder.params.open-gop=False
window.5.encoder.params.sliced-threads=False
window.5.encoder.params.threads=12
window.5.encoder.params.vfr-input=False
window.5.encoder.pixels_per_second=159132896
window.5.encoder.preset=veryfast
window.5.encoder.profile=baseline
window.5.encoder.quality=58
window.5.encoder.source=video
window.5.encoder.speed=82
window.5.encoder.src_format=YUV420P
window.5.encoder.total_time_ms=13
window.5.encoder.tune=film
window.5.encoder.version=148
window.5.encoder.width=640

Still TODO:


Thu, 07 Jun 2018 11:33:55 GMT - Antoine Martin:

why am I seeing many delayed frames?

I was and now I'm not. No idea why.

some corrupted frames too?

I believe those were jpeg frames, there were more of those because of this bug: r19579

Just like #1816, this is ready for testing. The quality is better than mpeg1, the bandwidth is slightly lower but the CPU usage is higher.


Tue, 12 Jun 2018 16:51:51 GMT - J. Max Mena: owner changed

Quick question:

How do I force it to use h264 over mpeg-1? In #1816 you mentioned that adding --video-encoders=ffmpeg will force mpeg-1. At least that's how I understood it. Running without that flag definitely has a quality increase. I also tried --video-encoders=x264 and nothing yelled at me so I assume (probably incorrectly) that it worked.


Tue, 12 Jun 2018 17:28:10 GMT - Antoine Martin: owner changed

How do I force it to use h264 over mpeg-1?

h264 will be used ahead of mpeg1 by default. If you really want to ensure that mpeg1 will not be used, add --video-encoders=x264,vpx You can always check what is actually being used using the paint boxes and -d compress.


Tue, 12 Jun 2018 20:06:48 GMT - J. Max Mena: owner changed

Okay that did it.

The h264 decoding looks really good - it's seriously impressive.

Is there anything in particular I should look for? As far as I can tell it's working great.

I'll use it some more today and see if I can't break something. But until then - it's pretty fantastic. I can actually play FTL (https://www.gog.com/game/faster_than_light) at a decent framerate (almost the same as the Python2 client) and is actually playable. That's totally awesome.


Tue, 12 Jun 2018 20:44:13 GMT - J. Max Mena: priority changed

Playing around with the HTML5 client some more I am fairly confident I've found a memory leak. If you open up the Chrome task manager (or whatever it's called) with shift+escape in Google Chrome, you can sort tabs by memory. Simply watching a video file with the HTML5 client causes the memory to increase steadily up to over 3GB after a few minutes. Disconnecting causes the memory to go away. Reconnecting with video decoders disabled does not cause the memory to climb.

EDIT: I should make it clear that I also started my server with --video-encoders=x264,vpx, so it's definitely running with h.264 and not mpeg-1.

I'm not sure what logs to get for you, but it's very obvious there's a memory leak with video enabled.


Wed, 13 Jun 2018 07:17:23 GMT - Antoine Martin: owner changed

I can only reproduce this by having the javascript console activated. In this case, the memory is used by the "GPU Process" which climbs at the same rate as the xpra html5 client. When I do see the memory leak, it is reproducible without h264 enabled, so it would not belong in this ticket. Note: there was a bug which prevented the server from suspending the video stream when the browser tab is not active, that's fixed in r19631. This does mean that the tab may have to be shown / active to reproduce the bug, as otherwise the screen updates stop.

There are lots of details missing, just to name a few:

Assuming that there is a leak without the debug console enabled, here are some links that may be relevant:


Wed, 13 Jun 2018 17:19:52 GMT - J. Max Mena:

Okay, in that case, I will follow up in a new ticket.


Wed, 20 Jun 2018 17:09:30 GMT - J. Max Mena: owner, priority changed

Since the memory leak isn't related to the h264 decoding, is there anything left on this ticket or should we close it?


Wed, 20 Jun 2018 17:24:07 GMT - Antoine Martin: status changed; resolution set

Okay, in that case, I will follow up in a new ticket.

The html5 client memleak ticket is #1879.

Since the memory leak isn't related to the h264 decoding, is there anything left on this ticket or should we close it?

I would have expected you guys to investigate bandwidth savings, CPU usage and such (this could be done using the automated tests). But this works well enough for me.


Sat, 24 Nov 2018 09:53:47 GMT - Antoine Martin:

Missed a glaring bug: #2060.


Sat, 23 Jan 2021 05:35:09 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1839