Xpra: Ticket #1617: detect silence and avoid sending empty packets

gstreamer's cutter element: Analyses the audio signal for periods of silence. The start and end of silence is signalled by bus messages named cutter - we could use this to tell the receiver to pause and resume.

Not sure if we want to / should wait for the pipeline to drain first.

See also: #1212.



Thu, 10 Aug 2017 11:29:10 GMT - Antoine Martin: status changed

Testing using gst-launch-1.0, I found that the opus codec just refuses to work with cutter in the pipeline

gst-launch-1.0 -m \
    pulsesrc ! queue ! cutter \
    opusenc !
    fakesink

All the other codecs+muxers work. (adding audioconvert after cutter as needed, see ENCODER_NEEDS_AUDIOCONVERT=("flacenc", "wavpackenc"))

Here's the error:

$ gst-launch-1.0 -m  pulsesrc ! queue ! cutter ! audioconvert ! opusenc !  fakesink
(...)
New clock: GstPulseSrcClock
Got message #41 from object "audiosrcringbuffer0" (stream-status): GstMessageStreamStatus, type=(GstStreamStatusType)GST_STREAM_STATUS_TYPE_ENTER, owner=(GstElement)"\(GstPulseSrc\)\ pulsesrc0", object=(GThread)NULL;
Got message #43 from element "opusenc0" (state-changed): GstMessageStateChanged, old-state=(GstState)GST_STATE_PAUSED, new-state=(GstState)GST_STATE_PLAYING, pending-state=(GstState)GST_STATE_VOID_PENDING;
Got message #44 from element "pulsesrc0" (latency): no message details
Redistribute latency...
Got message #45 from element "audioconvert0" (state-changed): GstMessageStateChanged, old-state=(GstState)GST_STATE_PAUSED, new-state=(GstState)GST_STATE_PLAYING, pending-state=(GstState)GST_STATE_VOID_PENDING;
Got message #46 from element "cutter0" (state-changed): GstMessageStateChanged, old-state=(GstState)GST_STATE_PAUSED, new-state=(GstState)GST_STATE_PLAYING, pending-state=(GstState)GST_STATE_VOID_PENDING;
Got message #48 from element "queue0" (state-changed): GstMessageStateChanged, old-state=(GstState)GST_STATE_PAUSED, new-state=(GstState)GST_STATE_PLAYING, pending-state=(GstState)GST_STATE_VOID_PENDING;
Got message #49 from element "pulsesrc0" (state-changed): GstMessageStateChanged, old-state=(GstState)GST_STATE_PAUSED, new-state=(GstState)GST_STATE_PLAYING, pending-state=(GstState)GST_STATE_VOID_PENDING;
Got message #55 from element "pulsesrc0" (error): GstMessageError, gerror=(GError)NULL, debug=(string)"gstbasesrc.c\(2939\):\ gst_base_src_loop\ \(\):\ /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0:\012streaming\ stopped\,\ reason\ not-negotiated\ \(-4\)", details=(structure)"details\,\ flow-return\=\(int\)-4\;";
ERROR: from element /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0:
streaming stopped, reason not-negotiated (-4)
(..)

Thu, 10 Aug 2017 12:52:01 GMT - Antoine Martin: owner, status changed

This seems to work well enough (with some caveats below), so added in r16675 (see commit for details).

Since this doesn't work with opus (for whatever reason - and maybe other codecs too..), maybe we should switch to another codec as default? (and file an upstream ticket to figure out why / how to fix this)

There was just one annoying limitation: the cutter element doesn't fire initially if the sound level starts already below the threshold. So as of r16676 we now play a short bell sample whenever a client starts forwarding sound. This ensures that the sound level goes above the threshold at least once.

We could also have added a level element in the pipeline and duplicate the "cutter" code to figure out what the initial state should be... but this wouldn't be 100% reliable.


With "-d gstreamer", when the stream goes near silent we see:

sound source on_message: <Gst.Message object at 0x7f1342639258 (GstMessage at 0x556c2f2d0200)>
sound source cutter message, above=False, min-timestamp=0, max-timestamp=63440000000
sound source cutter: skipping buffer with pts=63450000000 (max-timestamp=63440000000)
(...)

Followed by constant "skipping buffer" messages.

And when the level rises again, we see:

(skipping buffer repeated)
sound source on_message: <Gst.Message object at 0x7f1342639258 (GstMessage at 0x556c2f2d0100)>
sound source cutter: skipping buffer with pts=64150000000 (max-timestamp=63440000000)
sound source cutter: skipping buffer with pts=64160000000 (max-timestamp=63440000000)
sound source cutter message, above=True, min-timestamp=64340000000, max-timestamp=0
sound source cutter: skipping buffer with pts=64170000000 (min-timestamp=64340000000)
(..)
sound source cutter: skipping buffer with pts=64330000000 (min-timestamp=64340000000)

@maxmylyn: ready for you to test (just don't use one of the broken codecs like opus). When no sound is being played, the sound subprocess should be mostly idle and there should not be any sound network traffic, the client sound buffer level should be stable near the minimum limit. When the volume reaches 2%, it should start forwarding again.

Note: there seems to be some problems (at least with the win32 builds) with some of the codecs, I'm not sure which ones are having problems because of this change and which ones have just bitrotted. flac, wav, wavpack and mp3 (on Linux) worked OK during testing. opus and vorbis did not..


Thu, 10 Aug 2017 19:28:39 GMT - J. Max Mena: owner changed

I'm seeing the same behavior as you noted (trunk r16677 Fedora 25 server and client) - flac/wav/wavpack/mp3 seem to behave nicely. Do you want me to go ahead and close this or do you want to do more work and get it working with opus and vorbis as well?

I'm assuming it's the latter.


Thu, 10 Aug 2017 23:14:41 GMT - J. Max Mena:

I guess I should clarify a bit. I'm fine with just switching the defaults, especially now that MP3 is a free codec. I think it would be nice if we could get this for all the sound codecs(okay I'm mostly hoping for Vorbis), but it's definitely not necessary if it's a ton of work.


Fri, 08 Sep 2017 10:15:18 GMT - Antoine Martin: status changed; resolution set

I've tried everything I could think of (re-ordering the pipeline, caps, etc) but I just cannot get "vorbis" or "opus" to play nice with the "cutter" element. So r16804 disables cutter with "vorbis" and makes "mp3" the default again. r16805 also disables it for "wavpackenc" and "avenc_aac".

I've asked the cutter with vorbis or opus: not-negotiated error

If and when we get an answer, I'll update the code accordingly if possible. Closing for now.


Tue, 12 Sep 2017 02:02:51 GMT - Antoine Martin:

XPRA_CUTTER_THRESHOLD=VALUE (defaults to 0.02=2%) can be used to control the maximum level required before cutter will kick in. Setting it to 0 will disable cutter.


Tue, 12 Sep 2017 14:13:48 GMT - Antoine Martin:

Related changes:


More information on the muxer problem:


Thu, 14 Sep 2017 15:53:45 GMT - Antoine Martin:

r16845 makes it possible to debug the cutter element specifically using the env var XPRA_SOUND_LOG_CUTTER=1, so we can see when the sound level goes above or below the threshold:

sound source cutter message, above=False, min-timestamp=0, max-timestamp=840000000
sound source cutter message, above=True, min-timestamp=13090000000, max-timestamp=0
sound source cutter message, above=False, min-timestamp=0, max-timestamp=19060000000
sound source cutter message, above=True, min-timestamp=22980000000, max-timestamp=0
sound source cutter message, above=False, min-timestamp=0, max-timestamp=28660000000
sound source cutter message, above=True, min-timestamp=30810000000, max-timestamp=0

Fri, 15 Sep 2017 05:10:31 GMT - Antoine Martin:

More testing and information in ticket:1638#comment:4.

wavenc was having problems with cutter, so r16847 adds it to the cutter blacklist.


Thu, 02 Nov 2017 12:20:20 GMT - Antoine Martin:

More fine tuning in r17285.


Tue, 11 Jun 2019 11:11:10 GMT - Antoine Martin:

See also #2325


Sat, 23 Jan 2021 05:29:14 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1617