nvenc produces invalid stream

[Antoine Martin]

Fairly easy to reproduce with glxgears.
This could be caused by some of the recent changes to the heuristics or maybe to nvenc itself:

• nvenc changelog
• r21050 damage events accounting
• r21047 region merging code changes

It would be nice to add a save-to-file feature to the video encoders.
Then we can see if the stream is corrupt or not.

(SAVE_VIDEO_FRAMES only records individual frames - which is also useful, for other use cases)

[Antoine Martin]

Here's what the client shows (seen with both h264 and h265 streams):

2018-11-20 21:30:45,463 client @13.635 Error: avcodec error decoding 9008 bytes of h265 data:
2018-11-20 21:30:45,464 client @13.635  'unknown format returned'
2018-11-20 21:30:45,464 client @13.635  frame 10
2018-11-20 21:30:45,464 client @13.636  decoder state:
2018-11-20 21:30:45,464 client @13.636          decoder_height = 320
2018-11-20 21:30:45,464 client @13.636                encoding = h265
2018-11-20 21:30:45,464 client @13.636              colorspace = YUV444P
2018-11-20 21:30:45,465 client @13.636       actual_colorspace = YUV444P
2018-11-20 21:30:45,465 client @13.636                  height = 300
2018-11-20 21:30:45,465 client @13.636           decoder_width = 320
2018-11-20 21:30:45,465 client @13.636                   width = 300
2018-11-20 21:30:45,465 client @13.637                 version = 58, 18, 100
2018-11-20 21:30:45,465 client @13.637                 formats = GBRP, RGB, YUV420P, BGRA, BGRX, ARGB, YUV422P, XRGB, YUV444P
2018-11-20 21:30:45,466 client @13.637                  frames = 10
2018-11-20 21:30:45,467 client @13.637                    type = avcodec
2018-11-20 21:30:45,467 client @13.637 Error: decode failed on 9008 bytes of h265 data
2018-11-20 21:30:45,467 client @13.637  300x300 pixels using avcodec
2018-11-20 21:30:45,467 client @13.638  frame options:
2018-11-20 21:30:45,467 client @13.638    quality=99
2018-11-20 21:30:45,468 client @13.638    encoding=h265
2018-11-20 21:30:45,468 client @13.638    pts=399
2018-11-20 21:30:45,468 client @13.638    frame=10
2018-11-20 21:30:45,468 client @13.638    csc=YUV444P
2018-11-20 21:30:45,468 client @13.638    speed=47

Sometimes with much higher frame numbers.

[Antoine Martin]

r21056 adds the ability to record video streams, ie:

XPRA_SAVE_VIDEO_STREAMS=1 xpra start ...

Playing back the streams with mplayer works fine, but does show some warnings for h265:

No pts value from demuxer to use for frame!
pts after filters MISSING
V:-9223372036854775808.0   0/  0 ??% ??% ??,?% 0 0

Because those streams lack any container, ffmpeg is unable to play them.

[Antoine Martin]

The missing pts may be a red herring: some h264 streams I've generated also trigger the mplayer warnings but play back fine in the client.

It might be useful to save the codec metadata with the stream (possibly all in the filename?), so we can reproduce the problem with our decoder code, but without requiring the whole client. ie: a simple command line tool would only need the file + codec, window size, csc mode.

[Antoine Martin]

ignore missing picture

[Antoine Martin]

The real cause is probably missing picture in access unit.

The patch above allows us to continue anyway by hacking the "options" dictionary and adding the "delayed" flag to it when we don't get any picture data from the decoder.
Effectively ignoring the error and pretending we knew this would be a "delayed" frame.

Proper solutions:

• disable h265 with nvenc (looks like this does not occur with h264)- easy
• make sure that nvenc always includes a picture - not sure how, maybe we need to turn off b-frames (support added recently apparently) or detect when they're used and add the "delayed" tag
• clean up the patch an continue when the data is missing..

[Antoine Martin]

Similar reports:

• ​rigaya: NVEnc
• ​ffmpeg 4718: Odd "missing picture in access unit" warnings after recent hevc parser changes

[Antoine Martin]

I can't reproduce this anymore.

If it happens again, we may want to add a way to export the nvenc encoder internal state as a client_options so we can then dump it when we encounter those broken frames.

[mjlbach]

I can reproduce this bug with glxgears on an Ubuntu 18.04 host/macOS client