xpra icon
Bug tracker and wiki

Opened 5 months ago

Last modified 3 months ago

#1816 new enhancement

decode mpeg1 in html5 client

Reported by: Antoine Martin Owned by: J. Max Mena
Priority: major Milestone: 2.4
Component: html5 Version: 2.2.x
Keywords: Cc:

Description

We are having problems enabling H264 video decoding (#1463), but maybe we can use mpeg1 instead via jsmpeg.
It's not going to be as efficient as H264, but still better than jpeg for video. And the API is clean and supports streaming via websockets: DECODE IT LIKE IT'S 1999

Attachments (2)

html5-mpeg1.patch (13.4 KB) - added by Antoine Martin 4 months ago.
mpeg1 encoder, decoder and stub html5 implementation
html5-mpeg1-v2.patch (3.3 KB) - added by Antoine Martin 4 months ago.
hook up mpeg1 decoding, not painting yet

Download all attachments as: .zip

Change History (12)

comment:1 Changed 5 months ago by Antoine Martin

Milestone: 3.02.4
Status: newassigned

Changed 4 months ago by Antoine Martin

Attachment: html5-mpeg1.patch added

mpeg1 encoder, decoder and stub html5 implementation

comment:2 Changed 4 months ago by Antoine Martin

Preparatory work:

  • r19296 minor fix for delayed frames (mpeg1 and mpeg2 always buffer one frame?)
  • r19297 support mpeg1 and mpeg2 decoding in python client

The patch attached implements encoding but still needs work:

  • it breaks muxing (fixable)
  • decoding is not actually implemented in the html5 client (couldn't figure out the jsmpeg api)

This may all be moot since H264 decoding seems to be fixed: #1839.
It could still be useful for very low powered device (ie: some arm boards?) where the decoding complexity of H264 is too much?

Last edited 4 months ago by Antoine Martin (previous) (diff)

Changed 4 months ago by Antoine Martin

Attachment: html5-mpeg1-v2.patch added

hook up mpeg1 decoding, not painting yet

comment:3 Changed 4 months ago by Antoine Martin

Updates:

  • r19306: test video encoders with two frames (since mpeg1 and mpeg2 delay one frame, they would error only on the second)
  • r19312: make muxer optional in ffmpeg encoder
  • r19313: implement mpeg1 and mpeg2 encoding, add python avcodec support for decoding mpeg2, and paint colors ("olive" and "lime")

For the html5 client, the patch above seems to decode OK, we just need to do something with the YUV data we get. Either re-use their WebGLRenderer or do software YUV-to-RGB as a POC. (and maybe ship a subset of jsmpeg1 - no need for the player and websocket bits - audio?)

Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:4 Changed 4 months ago by Antoine Martin

html5 mpeg1 decoder added in r19317. (we won't be doing mpeg2: jsmpeg : Support MPEG2?)

Still TODO:

  • check CPU load and bandwidth: compare it with h264 (#1839 - once asm.js optimization is fixed there) so we can use weights to choose the most appropriate video encoder
  • trim jsmpeg.js file
  • re-use the webgl checks to enable the gl renderer in other places?

Note: the "mpeg1" video encoding is controlled by the "Video" option in the advanced connection options. This enables both h264 and mpeg1. To use only mpeg1, set --video-encoders=ffmpeg on the server side.

Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:5 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to J. Max Mena
Status: assignednew

Updates:

  • r19577: epic fail - was causing a massive drop in framerate (affected trunk only - glad I didn't backport this!)
  • r19578: disable webgl rendering: I've seen unrecoverable webgl errors, so we can't rely on it
  • r19579: this refresh bug caused unnecessary refresh and jerky rendering
  • not going to trim the jsmpeg file, which is small enough already and not much to save anyway

Requirements:

  • enable "Video" in html5 connection advanced options
  • ffmpeg-xpra version 4.0-2 or later
  • best to use --video-encoders=ffmpeg to ensure only mpeg1 is used for testing

Initial testing shows that my decent CPU can render 1080p at a decent framerate (close to 25fps), each frame taking about 10 to 50KB - meaning around 2 to 10Mbps, roughly.
Note: mpeg1 always buffers one frame, which is fine for sending video.

@maxmylyn: ready for testing. worth comparing with #1839

comment:6 Changed 3 months ago by J. Max Mena

Owner: changed from J. Max Mena to Antoine Martin

It works great - except native decoding seems to be extremely choppy and spews lots of tracebacks on both my Fedora and Windows8.1 boxes.

Is there anything in particular I should be looking at?

comment:7 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to J. Max Mena

It works great - except native decoding seems to be extremely choppy and spews lots of tracebacks on both my Fedora and Windows8.1 boxes.

What tracebacks? Always include all tracebacks.

In any case, mpeg1 is not meant to be used by the python client, it is designed for the html5 client.

comment:8 Changed 3 months ago by J. Max Mena

Owner: changed from J. Max Mena to Antoine Martin

For reference I'm running Trunk r19621 on Fedora 28 with Chrome in Fedora and Windows8.1.

On the client side I see the following:

Client.js:1 error painting mpeg4+mp4 DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
    at XpraWindow._push_video_buffers (http://192.168.1.81:10000/js/Window.js:1:18041)
    at XpraWindow.do_paint (http://192.168.1.81:10000/js/Window.js:1:25593)
    at XpraWindow.may_paint_now (http://192.168.1.81:10000/js/Window.js:1:20976)
    at XpraWindow.paint (http://192.168.1.81:10000/js/Window.js:1:20603)
    at XpraClient._process_draw_queue (http://192.168.1.81:10000/js/Client.js:2:15980)
    at XpraClient._process_draw (http://192.168.1.81:10000/js/Client.js:2:14532)
    at XpraProtocolWorkerHost.XpraClient._route_packet [as packet_handler] (http://192.168.1.81:10000/js/Client.js:1:9239)
    at Worker.<anonymous> (http://192.168.1.81:10000/js/Protocol.js:1:707)
XpraClient.error @ Client.js:1
XpraWindow.error @ Window.js:1
paint_error @ Window.js:1
XpraWindow.do_paint @ Window.js:1
XpraWindow.may_paint_now @ Window.js:1
XpraWindow.paint @ Window.js:1
XpraClient._process_draw_queue @ Client.js:2
XpraClient._process_draw @ Client.js:2
XpraClient._route_packet @ Client.js:1
(anonymous) @ Protocol.js:1
Window.js:1 Uncaught (in promise) DOMException: The element has no supported sources.

On the server side I see the following

2018-06-12 10:33:35,576 client video source buffer error
2018-06-12 10:33:35,576 client video error
2018-06-12 10:33:35,698 client error painting mpeg4+mp4 InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
2018-06-12 10:33:35,713 Warning: client decoding error:
2018-06-12 10:33:35,715  InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
2018-06-12 10:33:36,345 client error painting mpeg4+mp4 InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
2018-06-12 10:33:36,347 Warning: client decoding error:
2018-06-12 10:33:36,347  InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
2018-06-12 10:33:36,360 client video media source EOS error: NotFoundError: Failed to execute 'removeSourceBuffer' on 'MediaSource': The SourceBuffer provided is not contained in this MediaSource.
2018-06-12 10:33:36,361 client video codec string: video/mp4; codecs="avc1.42C01E" for mpeg4+mp4 profile 'baseline', level '3.0'
2018-06-12 10:33:36,362 client video media source open
2018-06-12 10:33:36,410 client video error

(in hindsight I definitely should have included the tracebacks)

comment:9 Changed 3 months ago by J. Max Mena

Also this error starts to print after a second or so:

Client.js:1 video error
XpraClient.error @ Client.js:1
XpraWindow.error @ Window.js:1
(anonymous) @ Window.js:1
error (async)
XpraWindow._init_video @ Window.js:1
XpraWindow.do_paint @ Window.js:1
XpraWindow.may_paint_now @ Window.js:1
XpraWindow.paint @ Window.js:1
XpraClient._process_draw_queue @ Client.js:2
XpraClient._process_draw @ Client.js:2
XpraClient._route_packet @ Client.js:1
(anonymous) @ Protocol.js:1
index.html:1 Uncaught (in promise) DOMException: The play() request was interrupted by a call to pause(). https://goo.gl/LdLk22

comment:10 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to J. Max Mena

Those stacktraces have nothing to do with this ticket, they're for native video #1463 - which is known to be broken, and doesn't use mpeg1.

Note: See TracTickets for help on using tickets.