xpra icon
Bug tracker and wiki

Opened 10 months ago

Last modified 2 months ago

#1775 assigned enhancement

html5 sound delay

Reported by: mviereck Owned by: Antoine Martin
Priority: minor Milestone: 2.5
Component: html5 Version: 2.2.x
Keywords: Cc:

Description

xpra v2.3-r18646

If viewing a video over html5 with sound forwarding, sometimes there are delays of a few seconds in the sound transfer. I've tested with firefox and chromium and several audio codecs, though not all combinations.

After the delay the sound goes on where it stopped. This leads to a mismatch of video and sound, especially annoying with mismatching dialogues of film characteres. Side effect: after terminating the video player, the sound goes on until the buffer is empty.

I assume the sound is buffered somewhere and not synchronized with xpra display transfer. Maybe an option to enforce video-sound synchronization that drops parts of sound if needed would make sense.

For myself it is not that important as I don't need that, just want to report the observation.

Change History (5)

comment:1 Changed 10 months ago by Antoine Martin

Milestone: 2.3
Status: newassigned

Thanks for reporting this.

sometimes there are delays of a few seconds in the sound transfer.

That should only happen if your connection has limited bandwidth, some jitter or if the browser is running slowly.

This code was added in ticket:1341#comment:6 (16 months ago), see also #845

The MediaSource (mozilla docs here: MediaSource API does have an attribute that may allow us to figure out if the browser is buffering too much (not sure how reliable implementations are going to be - ie: Safari will probably just crash if we ask): media.buffered: The buffered attribute must return a new static normalized TimeRanges object that represents the ranges of the media resource, if any, that the user agent has buffered, at the time the attribute is evaluated. Users agents must accurately determine the ranges available, even for media streams where this can only be determined by tedious inspection..

Last edited 9 months ago by Antoine Martin (previous) (diff)

comment:2 Changed 9 months ago by Antoine Martin

Oh, what a mess. I'm pretty sure things have changed since the HTML5 mediasource code was merged.

My browsers seem to be playing the stream too fast and changing the sample rate at the gstreamer source pipeline doesn't make any difference. Neither does changing the playback rate in javascript.

  • Firefox deals with this better: it tries to keep the buffer level at around 100 to 200ms, stuttering a bit when needed. If it didn't play too fast, and if we also had html5 video (#1463), we could attempt av-sync (#835).
  • Chrome just stops playing and buffers 3 seconds whenever the level drops below 100ms... but then it also buffers more, somewhere else, because when we stop playing audio, it keeps on playing for much longer than what the buffer shows.

PS: chrome 62+ now supports flac+mp4, we could enable it with a version check.

Here's a patch which dumps information to the javascript console:

--- html5/js/Client.js	(revision 18706)
+++ html5/js/Client.js	(working copy)
@@ -2308,6 +2308,29 @@
 XpraClient.prototype.push_audio_buffer = function(buf) {
 	if (this.audio_framework=="mediasource") {
 		this.audio_source_buffer.appendBuffer(buf);
+		var b = this.audio_source_buffer.buffered;
+		if (b && b.length==1) {
+			/*console.log("buffered=", b.length, "timestampOffset=", this.audio_source_buffer.timestampOffset);
+			for (var i=0; i<b.length;i++) {
+				console.log("buffered[", i, "]=", b.start(i), b.end(i));
+			}
+			var p = this.audio.played;
+			console.log("played=", p.length);
+			for (var i=0; i<p.length;i++) {
+				console.log("played[", i, "]=", p.start(i), p.end(i));
+			}*/
+			var e = b.end(0)
+			var buf_size = e - this.audio.currentTime;
+			console.log("buffer size=", Math.round(1000*buf_size), "ms, currentTime=", this.audio.currentTime);
+			/*
+			if (this.audio.readyState>=3 && buf_size<0.2) {
+				this.audio.playbackRate = Math.max(0.9, Math.min(this.audio.playbackRate, 1)-0.01);
+			}
+			if (this.audio.readyState>=3 && buf_size>0.8) {
+				this.audio.playbackRate = buf_size/0.8;
+			}*/
+			console.log("state=", this.audio.readyState, "network state=", this.audio.networkState, "playbackRate=", this.audio.playbackRate);
+		}
 	}
 	else {
 		this.audio_aurora_ctx.asset.source._on_data(buf);

comment:3 Changed 7 months ago by Antoine Martin

Milestone: 2.32.4

comment:4 Changed 7 months ago by Antoine Martin

The solution may be found here: Decode it like it's 1999: WebAudio for Live Streaming: ... With this, you can string very short PCM buffers together without any artefacts. You only have to calculate the start time for the next buffer by continuously adding the duration of all previous ones.

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:5 Changed 2 months ago by Antoine Martin

Milestone: 2.42.5

Switching codecs makes no difference.
To make matters worse, it seems that you can try the exact same settings and get completely different results.
I give up, for now, browser audio is too much of a mess at the moment.

Note: See TracTickets for help on using tickets.