Xpra: Ticket #1699: pass hints to the encoder

See ticket:1694#comment:4 and ticket:999#comment:24

We can add some hints to the window-model to try to help the picture encoders make better decisions. For example:

etc

The webp encoder (#1694) can use this hint, we can also use this hint for selecting video mode.



Tue, 28 Nov 2017 16:30:29 GMT - Antoine Martin: status, description changed


Sat, 16 Dec 2017 07:49:43 GMT - Antoine Martin: owner, status changed

Added in r17665, see commit message for details. An xterm shows up as:

$ xpra info | grep content-type
window.1.content-type=text

A Firefox window correctly shows up as:

$ xpra info | grep content-type
window.1.content-type=browser

And the webp encoder now shows details on "preset" and "image hint":

webp.compress(XImageWrapper(BGRX: 0, 1317, 1513, 13), 1513, 13, False, browser) buf=0x7f072c8624f4
webp.compress config: lossless=0    , quality= 40, method=0, alpha=  0,  0,  0, preset=text    , image hint=graph

@maxmylyn: this is not finished yet, but already a good time to hand this over to you:

The coarse webp tunings using "preset" and "image hint" may not be enough for our needs - in which case we'll need to enhance the webp encoder logic, see libwebp/enc/config.c for what those preset values do.


Thu, 04 Jan 2018 19:08:18 GMT - J. Max Mena:

Okay part one of comment:2 is done - as far as I can tell most major applications I've tried show up correctly (even a couple Electron applications). I'll write a script to test the webp encoder today or tomorrow depending on when I get to it.


Thu, 08 Mar 2018 22:37:51 GMT - J. Max Mena:

Okay, I've finally gotten around to the second part of comment:2 after much reading (mostly learning Bash, to be honest) I cobbled together a not very pretty bash script that will loop through all the presets and quality from 10 to 100 (inclusive) in increments of 10. It will save the outputted file and logs from said compression into subfolders with the name of the preset. If anyone would like to reproduce my metrics, I'll attach the .sh file. You can replace the cwebp......."image.png" lines with whatever image inputs you'd like - just make sure to change the log file name accordingly. It's not pretty nor is it configurable, but it's for a fairly single-use case. Yes it could have been prettier, yes it could have accepted inputs, but at this point I don't care. I'm sorry in advance.

(There's also a second bash script that just does lossless that I'll also attach)

I created three test images - two that are stills from the Documentary "Inside Job" - I chose this one because I watched it again last week on a whim, had the 1080p version available on my Emby Server, and it's also available free on archive.org or Vimeo - one is just a picture of some scenery from Iceland, and the other is the title text. The third image is a Wikipedia page with a picture on the top right. So we got a nice well-rounded three images to work with. They're titled "Video.png", "Image and Text.png", and "Video Large Text.png"

Also of use: https://developers.google.com/speed/webp/docs/using

NOTE: -v adds more verbose logging of the encoding itself - including encoding time!

There's a lot more flags we can use - but I think implementing the presets and quality level is a great first step.

Anyways, I'll throw together a quick spreadsheet with relevant info.


Fri, 09 Mar 2018 03:09:50 GMT - J. Max Mena: attachment set

webp compression script


Fri, 09 Mar 2018 05:07:03 GMT - Antoine Martin:

r18670 normalizes the webp quality lower. Please use a wiki table for the data, and maybe also attach the sample pictures so we can reproduce the data later, and again after webp library updates. (the current version in Fedora 27 is 0.6.1)


Fri, 09 Mar 2018 22:41:43 GMT - J. Max Mena:

Okay I managed to comb through the data and put together a nice spreadsheet. Interestingly, the photo preset consistently has a notably higher encoding time than the others.

I'll attach a .csv of the data, and a .xlsx with charts (made it in Google Drive). Yes I know proprietary formats are bad, but the charts make it pretty obvious what is going on at a glance.

I'll write up a Wiki page shortly. I'll put it in wiki/WebpEncodingData.


Fri, 09 Mar 2018 22:42:33 GMT - J. Max Mena: attachment set


Fri, 09 Mar 2018 22:42:41 GMT - J. Max Mena: attachment set


Fri, 09 Mar 2018 22:46:38 GMT - J. Max Mena:

Actually, it turns out you can export Google Sheet charts as a .png, so I'll post those instead. Also the original images as you asked.


Fri, 09 Mar 2018 22:47:39 GMT - J. Max Mena: attachment set

Chart of data from Video.png


Fri, 09 Mar 2018 22:48:46 GMT - J. Max Mena: attachment set

Chart of data from Video Large Text.png


Fri, 09 Mar 2018 22:49:02 GMT - J. Max Mena: attachment set

Chart of data from Image and Text.png


Fri, 09 Mar 2018 22:49:47 GMT - J. Max Mena: attachment set

The original screenshot of the video


Fri, 09 Mar 2018 23:02:22 GMT - J. Max Mena: attachment set

The same as the other shell script, but lossless and I fixed a typo


Sat, 10 Mar 2018 02:31:57 GMT - Antoine Martin:

The outlier is the "photo" preset, but even then it doesn't take that much more time to encode (~10%). So, which presets are useful for which types of content? Do they make a noticeable difference? I also expect the lossless results to look quite different, judging from the API usage.


Thu, 29 Mar 2018 17:11:51 GMT - J. Max Mena: owner changed

Okay I finally got around to visually inspecting the images. I'd attach them here, but all together they're ~40mb even compressed with gzip. It'd be better if you recreated them on your own using the bash script I attached.

Before I get into the subjective, I'll point out that the lossless usually performs similar to 50-70% quality for each preset. I'm honestly not sure if that's a good or a bad thing, but it's worth noting that lossless will perform better than higher quality lossy encodings.

On to the subjective:

From what I can see, looking at the worst cases, icon performs the worst in most cases. It makes text readable with no issue even on the lowest quality, so there's that.

For text-only content, text, unsurprisingly, looks the best. However for regular images, text looks subjectively worse than any of the other encodings - it has a hard time with color gradients - in particular trees look rather dreadful on the lower qualities.

Other than that - I can't really tell that much of a difference between the others - at a glance they all kind of look the same to me. I noticed that picture tens to get the colors right most of the time, so there's that - the others can have the lossy color chunks, but picture tends to get the colors more spot on while ignoring the lines.

From my subjective look - I'd say go with text for text only situations, and then with whatever performs the best for anything else.


Sat, 31 Mar 2018 14:39:47 GMT - Antoine Martin: owner changed

Before I get into the subjective, I'll point out that the lossless usually performs similar to 50-70% quality for each preset.

"performance" - I assume that you mean compression latency only here? Do you have the resulting file size data? We care about both compression latency and compression ratio. The lossless mode has completely different performance characteristics: the "quality" setting is then used to drive compression effort, at least when using the API. (more or less what we usually call "speed" in xpra). A lossless picture is worth spending a bit more time to compress since we can then skip the whole auto-refresh logic and the eventual lossless refresh that this may trigger later. But not if the resulting compressed data is too much bigger. We currently switch to lossless mode when the compression request has a quality value higher than XPRA_WEBP_LOSSLESS_THRESHOLD which defaults to 75.

icon performs the worst in most cases

We only use the icon as default preset for very small rectangles, I've made this even stricter in r18926.

For text-only content, text, unsurprisingly, looks the best. However for regular images, text looks subjectively worse than any of the other encodings - it has a hard time with color gradients - in particular trees look rather dreadful on the lower qualities.

That's annoying, as I've made "text" the default. It's not as bad as it sounds because the lossless auto refresh should always eventually fix that (and video is handled separately) - should we change it? To what?

I'd say go with text for text only situations, and then with whatever performs the best for anything else.

Most of the time, unless we detect the application or if it supplies us with a hint, we don't know what we're dealing with so "text" seems like the conservative choice: making text unreadable is worse than color changes I would think?

Minor related fix in r18927.


Tue, 03 Apr 2018 17:51:51 GMT - J. Max Mena: owner changed

"performance" - I assume that you mean compression latency only here? Do you have the resulting file size data?

We care about both compression latency and compression ratio.

Right I should specify, my bad.

By performance there I meant compression time - lossless compresses fairly quickly.

As for file size, they're all very similar - here's the table for the default encoding of the image "Image and Text.png":

Quality Filesize in Bytes
original 851739
10 144804
20 173878
30 198934
40 221090
50 237386
60 255202
70 271838
80 314068
90 397692
100 556888
lossless 281438

For all the presets, I see a very similar trend - a notable increase in file-size as quality increases. However, Lossless seems to hover around the 50-70 quality. Which mirrors the encoding times as reported in the spreadsheets I attached.


That's annoying, as I've made "text" the default. It's not as bad as it sounds because the lossless auto refresh should always eventually fix that (and video is handled separately) - should we change it? To what?

picture performs similarly and has an acceptable level of quality for text even at the lowest levels of quality. But,

Most of the time, unless we detect the application or if it supplies us with a hint, we don't know what we're dealing with so "text" seems like the conservative choice: making text unreadable is worse than color changes I would think?

I concur with you - leaving text as the default is a good conservative choice, at least as long as we aren't receiving hints as to the content type for a particular application.


Tue, 03 Apr 2018 18:21:34 GMT - Antoine Martin: status changed; resolution set

OK then I think we're good, we're not using compression levels above 75%. If I had more time, I would benchmark the lossless compression with different "speed" values, but this will do for now.


Sun, 09 Sep 2018 04:35:22 GMT - Antoine Martin:

Follow up: #1950


Sat, 23 Jan 2021 05:31:24 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1699