xpra icon
Bug tracker and wiki

Opened 20 months ago

Last modified 4 weeks ago

#1775 assigned enhancement

html5 sound delay

Reported by: mviereck Owned by: Antoine Martin
Priority: minor Milestone: 4.0
Component: html5 Version: 2.2.x
Keywords: Cc:

Description

xpra v2.3-r18646

If viewing a video over html5 with sound forwarding, sometimes there are delays of a few seconds in the sound transfer. I've tested with firefox and chromium and several audio codecs, though not all combinations.

After the delay the sound goes on where it stopped. This leads to a mismatch of video and sound, especially annoying with mismatching dialogues of film characteres. Side effect: after terminating the video player, the sound goes on until the buffer is empty.

I assume the sound is buffered somewhere and not synchronized with xpra display transfer. Maybe an option to enforce video-sound synchronization that drops parts of sound if needed would make sense.

For myself it is not that important as I don't need that, just want to report the observation.

Change History (9)

comment:1 Changed 20 months ago by Antoine Martin

Milestone: 2.3
Status: newassigned

Thanks for reporting this.

sometimes there are delays of a few seconds in the sound transfer.

That should only happen if your connection has limited bandwidth, some jitter or if the browser is running slowly.

This code was added in ticket:1341#comment:6 (16 months ago), see also #845

The MediaSource (mozilla docs here: MediaSource API does have an attribute that may allow us to figure out if the browser is buffering too much (not sure how reliable implementations are going to be - ie: Safari will probably just crash if we ask): media.buffered: The buffered attribute must return a new static normalized TimeRanges object that represents the ranges of the media resource, if any, that the user agent has buffered, at the time the attribute is evaluated. Users agents must accurately determine the ranges available, even for media streams where this can only be determined by tedious inspection..

Last edited 19 months ago by Antoine Martin (previous) (diff)

comment:2 Changed 19 months ago by Antoine Martin

Oh, what a mess. I'm pretty sure things have changed since the HTML5 mediasource code was merged.

My browsers seem to be playing the stream too fast and changing the sample rate at the gstreamer source pipeline doesn't make any difference. Neither does changing the playback rate in javascript.

  • Firefox deals with this better: it tries to keep the buffer level at around 100 to 200ms, stuttering a bit when needed. If it didn't play too fast, and if we also had html5 video (#1463), we could attempt av-sync (#835).
  • Chrome just stops playing and buffers 3 seconds whenever the level drops below 100ms... but then it also buffers more, somewhere else, because when we stop playing audio, it keeps on playing for much longer than what the buffer shows.

PS: chrome 62+ now supports flac+mp4, we could enable it with a version check.

Here's a patch which dumps information to the javascript console:

--- html5/js/Client.js	(revision 18706)
+++ html5/js/Client.js	(working copy)
@@ -2308,6 +2308,29 @@
 XpraClient.prototype.push_audio_buffer = function(buf) {
 	if (this.audio_framework=="mediasource") {
 		this.audio_source_buffer.appendBuffer(buf);
+		var b = this.audio_source_buffer.buffered;
+		if (b && b.length==1) {
+			/*console.log("buffered=", b.length, "timestampOffset=", this.audio_source_buffer.timestampOffset);
+			for (var i=0; i<b.length;i++) {
+				console.log("buffered[", i, "]=", b.start(i), b.end(i));
+			}
+			var p = this.audio.played;
+			console.log("played=", p.length);
+			for (var i=0; i<p.length;i++) {
+				console.log("played[", i, "]=", p.start(i), p.end(i));
+			}*/
+			var e = b.end(0)
+			var buf_size = e - this.audio.currentTime;
+			console.log("buffer size=", Math.round(1000*buf_size), "ms, currentTime=", this.audio.currentTime);
+			/*
+			if (this.audio.readyState>=3 && buf_size<0.2) {
+				this.audio.playbackRate = Math.max(0.9, Math.min(this.audio.playbackRate, 1)-0.01);
+			}
+			if (this.audio.readyState>=3 && buf_size>0.8) {
+				this.audio.playbackRate = buf_size/0.8;
+			}*/
+			console.log("state=", this.audio.readyState, "network state=", this.audio.networkState, "playbackRate=", this.audio.playbackRate);
+		}
 	}
 	else {
 		this.audio_aurora_ctx.asset.source._on_data(buf);

comment:3 Changed 18 months ago by Antoine Martin

Milestone: 2.32.4

comment:4 Changed 17 months ago by Antoine Martin

The solution may be found here: Decode it like it's 1999: WebAudio for Live Streaming: ... With this, you can string very short PCM buffers together without any artefacts. You only have to calculate the start time for the next buffer by continuously adding the duration of all previous ones.

Last edited 12 months ago by Antoine Martin (previous) (diff)

comment:5 Changed 12 months ago by Antoine Martin

Milestone: 2.42.5

Switching codecs makes no difference.
To make matters worse, it seems that you can try the exact same settings and get completely different results.
I give up, for now, browser audio is too much of a mess at the moment.

comment:6 Changed 8 months ago by Antoine Martin

Milestone: 2.53.0

See also #2165

Last edited 8 months ago by Antoine Martin (previous) (diff)

comment:7 Changed 4 weeks ago by Antoine Martin

  • r23790 adds some "audio" debug logging
  • r23794 'concat' testing option for audio buffers
  • r23795 support data bundled with the start-of-stream marker
  • r23796 add 'mp3+id3v2', r23797 for html5 client
  • r23798 aurora debug logging
  • r23799 try harder to close the aurora audio context cleanly

So, it seems that we have no way of knowing how much is buffered internally by the browser with MSE? And that changing the playback rate does not allow us to get back to a small buffer size.. And we also have the problem of #2322.
So Media Source Extensions aren't very helpful at all and I thought I would try to go back to aurora decoding to see if we can get better buffering behaviour, it turns out that only WAV playback works (which is why the other codecs were disabled in r15732) and that we also can't always start an audiocontext when loading the page (no autostart), and that when it fails, it fails with a warning in the javascript console only.. We would need to get to the AudioContext inside aurora to see if the state is paused.
On the plus side, AudioContext.baseLatency could be useful. The AudioContextOptions.latencyHint default value is interactive: meaning the browser should try to use the lowest possible and reliable latency it can.
See also AudioContext.getOutputTimestamp
WebAudio: Add AudioContext.getOutputTimestamp() method: The AudioContext.getOutputTimestamp() method helps to synchronize DOM time and AudioContext time values. It is used to estimate DOMHighResTimeStamp value of the audio output stream position for the given AudioContext.currentTime value, or do the opposite: estimate AudioContext.currentTime value of the audio output stream position for the given DOMHighResTimeStamp value
AudioContext: baseLatency and outputLatency
HTML5 audio streaming: precisely measure latency?
How to live-stream raw PCM through HTML5 audio element: You can use the ScriptProcessorNode in the Web Audio API to play arbitrary PCM samples at very low latency. Then, you can manage the buffering however you want. - now superseded by AudioWorkletNode
PCM to WAV in Javascript

Also, using XPRA_SAVE_TO_FILE=basename to save the mp3 / flac data to file, it looks like the mp3js code (tested using http://audiocogs.org/codecs/mp3/) needs the new id3v2mux muxer element added in r23796.
When mp3 decoding does work is when we buffer enough chunks before submitting them to the decoder, then it still errors out after consuming them..
And lately, even wav seems to be broken!?

Last edited 4 weeks ago by Antoine Martin (previous) (diff)

comment:8 Changed 4 weeks ago by Antoine Martin

This could well be aurora bug(s):

comment:9 Changed 4 weeks ago by Antoine Martin

Milestone: 3.04.0

FWIW: increasing MIN_START_BUFFERS to 100 allows the "legacy: mp3" codec to start playing, but it stops shortly after. None of the fixes linked from comment:8 make any difference.

Beats me.

Note: See TracTickets for help on using tickets.