Xpra: Ticket #1370: Improved timestamps on AV packets

If the server send timestamps sound-data and draw packets that represent not just when the data comes out of the encoder, but also when the raw data goes into the encoder. this information could be used by the client to better track how far out of sync audio is compared to video by comparing when the data went into the respective encoders.



Wed, 23 Nov 2016 17:28:04 GMT - Antoine Martin: priority, status, component changed; milestone set

The "timestamps" we currently get from the gstreamer pipeline are already from the input element AFAICT (TBC), adding a "do-timestamp=True" to the "pulsesrc" doesn't change that. The problem is that they start from 0, with no way for us to convert them to absolute time.

Looks like we'll need to

I've tried:

sysclock = gst.SystemClock.obtain()
self.pipeline.use_clock(sysclock)

But that didn't make any difference.

Links:

I think I'll need to ask the gstreamer devs. In any case, this is 2.0 material, let's remove gstreamer 0.10 first.


Wed, 15 Feb 2017 11:20:46 GMT - Antoine Martin:

I've updated the doc links which had bit-rotted, and asked the absolute buffer timestamp from src element.


Tue, 21 Feb 2017 12:34:40 GMT - Antoine Martin: status, milestone changed

Judging by the lack of response, there is no easy way to do this. We'll have to write a custom gstreamer audio filter element and expose the timestamp delta as a property we can query from our capture pipeline.

Pointers:


Wed, 31 May 2017 17:34:54 GMT - Antoine Martin: attachment set

timestamp plugin


Wed, 31 May 2017 17:35:54 GMT - Antoine Martin: attachment set

header file for timestamp plugin


Wed, 31 May 2017 17:37:21 GMT - Antoine Martin: attachment set

use the timestamp plugin to get a real monotonic absolute timestamp we can also use for video frames


Wed, 31 May 2017 17:42:12 GMT - Antoine Martin: status changed

To use the new plugin attached:

gcc -I. -I.. \
    -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include \
    -pthread -Wall -g -O2 -Wall  -c gsttimestamp.c  -fPIC -DPIC -o gsttimestamp.o
gcc -shared  -fPIC -DPIC  gsttimestamp.o  \
    -lgstbase-1.0 -lgstcontroller-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 \
    -pthread -g -O2   -pthread -Wl,-soname -Wl,libgsttimestamp.so  -o libgsttimestamp.so
sudo cp libgsttimestamp.so /usr/lib64/gstreamer-1.0/

Then with the xpra patch above, we expose extra attributes in the audio packet metadata:

Still TODO:


Thu, 01 Jun 2017 10:18:08 GMT - Antoine Martin:

The timestamp value is sent as part of the audio and video metadata under the key "ts". (the existing "timestamp" is still present for audio packets - but this is an absolute value which cannot be reconciled with the video packets metadata)

Note: this is how long it takes from the moment the audio buffer is captured from the source element (typically "pulsesrc") until we receive it in a compressed form in our buffer capture element ("appsink"). It does not include the time it takes to capture from the sound card (usually very low anyway), or the time it takes to forward this buffer through our subprocess wrapper layer back to the server process (also quite low)


Thu, 01 Jun 2017 12:50:42 GMT - Antoine Martin: owner, status changed

Beta builds with those changes are available here http://xpra.org/beta.


Mon, 17 Jul 2017 09:25:45 GMT - Antoine Martin: milestone changed

(edit milestone)


Thu, 20 Jul 2017 07:18:20 GMT - Antoine Martin: priority changed

this is an important feature - please test


Fri, 08 Sep 2017 10:38:42 GMT - Antoine Martin: status changed; resolution set

Not heard back, closing.


Sat, 23 Jan 2021 05:22:23 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1370