libva accelerated encoding

[Antoine Martin]

Split from #370, see also #202 for the decoding side.

The only hardware supported by libva for accelerated encoding are some Intel chips.

Pointers:

• ​libva
• ​qemu patch (good starting point / example)

Note: libva can encode to other formats (avc, mpeg2, ..) and maybe this can be useful in the future.

[Antoine Martin]

Raising, Intel's ​Valley Vista makes this much more interesting. Awaiting hardware to take this on.

[Antoine Martin]

stub implementation

[Antoine Martin]

Spotted: ​H.264 Encoding On Skylake Gets A Big Performance Boost, details here: ​Add Low power, High performance h264 encoder support.

[Antoine Martin]

Could be used for 10-bit colour too, see #909.

[Antoine Martin]

As per ​Fedora 28 Planning For VA-API 1.0 Support, Libva 2.0 was released last October with H.264 FEI support in its API, deprecating older parts of the API, fixing a race condition with the Wayland support, renaming some parts of the API, improving the logging capabilities, and various other changes. Libva 2.0 broke API/ABI compatibility with older versions of this Intel-developed Video Acceleration API.

We want to target libva 2.0, so let's wait for Fedora 28 beta.

OBS does it through ffmpeg: ​Add linux VAAPI h.264 encoding support - I'm not sure we should do the same: ideally we can reduce the dependency on ffmpeg, not increase it.

[Antoine Martin]

Milestone renamed

[Antoine Martin]

See also #1666

• ​https://trac.ffmpeg.org/wiki/Hardware/VAAPI
• ​https://wiki.libav.org/Hardware/vaapi
• ​ffmpeg VAAPI-encoders
• ​hwaccel : Platform API Availability
• example here: ​vaapi_encode.c
• ​Using the VAAPI with ffmpeg
• ​h264encode.c

[Antoine Martin]

Contrary to what I said in ticket:1666#comment:6, I can't get vaapi to encode anything with my nvidia card, but it does work well with the Intel iGPU.

Mostly working as of:

• r26720 (+r26723 + r26730 fixups) libyuv support for NV12, since we need this pixel format to feed vaapi
• r26721 use vaapi for the codecs that are available
• r26722 enable vaapi in ffmpeg rpm

With the default quality setting, the performance is quite good.

Still TODO:

• add NV12 support to csc_swscale so Debian users aren't left behind (why don't they just package libyuv??)
• fix bugs (ie: color problems, client decoding errors, etc)
• test on AMD
• test more codecs, adapt encoder scoring
• assuming this works well enough, backport to v3.1

[Antoine Martin]

Updates:

• r26725: NV12 support for swscale - also needed for when we want to downscale video before compressing since libyuv can't do that yet

New bug when testing with glxgears: do_check_pipeline: csc swscale(BGRX 300x300 - NV12 300x300) is closed - why is it closing?

New TODO: do scaling with libyuv before converting to NV12?

[Antoine Martin]

New bug when testing with glxgears...

This bug affected csc_swscale, which is used by the distros that don't have csc_libyuv (ie: Debian and Ubuntu for sure, maybe others), it was introduced in r8125 (v0.15.x in 2014!), just fixed now in r26734.
This caused video pipelines with csc_swscale to constantly recycle. Amazing that this was not spotted earlier. (I do have libyuv installed almost everywhere, so there is that)

[Antoine Martin]

Updates:

• r26749 hide warnings and messages during initialization / probing
• r26751 vaapi is Linux-only
• r26750 + r26752 fast path logging callback, XPRA_LIBAV_TRACE=1 env var to enable full trace logging
• r26753 simplify win32 sanity checks
• r26754 type safe access to encoding options
• r26755 set more codec specific options
• r26756 set profile correctly (support html5 client - which needs baseline)
• r26757 score h264 and hevc higher than mpeg2 by default

Notes about this updated enc_ffmpeg codec:

• we use a fixed quality which is set when the encoder is initialized
• could also be used for encoding h264, but somehow the html5 client then fails to decode the stream (problem with nals? or different settings from enc_x264?)
• vp9 and hevc cause the process to exit! (ouch! so they're not enabled..)
• ​pyav is neat - could this replace the cython glue? (and keep the zerocopy?)

This will do for now.
More testing needed, especially with an AMD card.