Xpra: Ticket #2113: Improving region detect on an OpenGL canvas

This ticket pursues the issue of ticket:2022#comment:47.

The desired behavior is for the video region to coincide with the OpenGL canvas throughout the entire run:

0x1000886 (has no name): ()  1582x854+3+29  +1874+6

I've attached a log.

The first rotation operation happens from 9:31 to 9:50

The second rotation happens until from the first log after the pause to the end.

In general, with logging enabled the whole application seems to be slow, but the underlying problem remains very detectable.

Mon, 21 Jan 2019 21:24:03 GMT - Nathan Hallquist: attachment set

Tue, 22 Jan 2019 04:54:27 GMT - Antoine Martin: attachment set

regularly log the client window's children

Tue, 22 Jan 2019 04:54:55 GMT - Antoine Martin: owner changed

If the window has a child window just for its canvas then we could use this geometry as a strong hint that this can be used as a region of interest during video region detection. Can you post the full xwininfo -root -tree for this application? And / or apply the patch above and see what comes out.

Note: unfortunately, this won't work for most browsers, chromium uses a single canvas for the entire window and does its own compositing:

     0x3600001 "SomeSite - Google Chrome": ("google-chrome-unstable" "Google-chrome-unstable")  2399x2000+343+60  +343+60
        2 children:
        0x3e00003 (has no name): ()  1x1+0+0  +343+60
        0x380000d (has no name): ()  2399x2000+0+0  +343+60

Tue, 22 Jan 2019 21:48:16 GMT - Nathan Hallquist:

I think this is a really good idea. Too bad it doesn't work for web browsers. I'm not familiar with how browser plugins work, but it would be nifty if there were a way to hint XPRA from a plugin.

I'm providing the xwininfo. If you want the patch, I'll do that. Let me know.

Here's the highlight:

0x100088c (has no name): ()  1370x776+3+29  +832+60

Here's the whole thing:

xwininfo: Window id: 0x311 (the root window) "Xpra"
  Root window id: 0x311 (the root window) "Xpra"
  Parent window id: 0x0 (none)
     21 children:
     0x40002b "Xpra-CorralWindow-0xc00022": ()  1196x752+8+90  +8+90
        1 child:
        0xc00022 "nathan@curry:~/lsprepost4.7_centos7": ("xterm" "XTerm")  1196x752+0+0  +8+90
           1 child:
           0xc0002f (has no name): ()  1196x752+0+0  +8+90
              1 child:
              0xc00034 (has no name): ()  14x752+-1+-1  +7+89
     0x40011e "Xpra-CorralWindow-0x10005de": ()  1526x899+829+31  +829+31
        1 child:
        0x10005de "LS-PrePost(R) V4.7.0 (Beta) - 27Dec2018-64bit ../issue/1.d3plot": ("lsprepost" "Lsprepost")  1526x899+0+0  +829+31
           6 children:
           0x10005ff (has no name): ()  673x23+689+853  +1518+884
           0x1000d62 (has no name): ()  680x17+4+856  +833+887
           0x10005f5 (has no name): ()  736x32+604+883  +1433+914
           0x1000d61 (has no name): ()  595x26+4+886  +833+917
           0x100088c (has no name): ()  1370x776+3+29  +832+60
           0x10005df (has no name): ()  1x1+-1+-1  +828+30
     0x40012b "Xpra-CorralWindow-0x1000b02": ()  455x136+3037+64  +3037+64
        1 child:
        0x1000b02 "Animate": ("lsprepost" "Lsprepost")  455x136+0+0  +3037+64
           6 children:
           0x1000d67 (has no name): ()  32x26+418+20  +3455+84
           0x1000d66 (has no name): ()  52x26+317+20  +3354+84
           0x1000d65 (has no name): ()  22x26+248+20  +3285+84
           0x1000d64 (has no name): ()  32x26+180+20  +3217+84
           0x1000d63 (has no name): ()  32x26+104+20  +3141+84
           0x1000b03 (has no name): ()  1x1+-1+-1  +3036+63
     0x1004340 "Configuration Settings": ("lsprepost" "Lsprepost")  994x711+0+0  +0+0
        9 children:
        0x10044d0 (has no name): ()  32x26+272+534  +272+534
        0x10044cf (has no name): ()  102x26+336+490  +336+490
        0x10044ce (has no name): ()  96x26+716+413  +716+413
        0x10044cd (has no name): ()  152x26+681+241  +681+241
        0x10044cc (has no name): ()  52x26+546+241  +546+241
        0x10044cb (has no name): ()  195x29+660+146  +660+146
        0x10044ca (has no name): ()  102x26+299+73  +299+73
        0x10045b1 (has no name): ()  148x660+5+5  +5+5
        0x1004341 (has no name): ()  1x1+-1+-1  +-1+-1
     0x100433d "lsprepost": ("lsprepost" "Lsprepost")  200x200+0+0  +0+0
        1 child:
        0x100433e (has no name): ()  1x1+-1+-1  +-1+-1
     0x10042ed "lsprepost": ("lsprepost" "Lsprepost")  227x154+1176+56  +1176+56
        1 child:
        0x10042ee (has no name): ()  1x1+-1+-1  +1175+55
     0x1000b1e "LS-PrePost 4.x Usage": ("lsprepost" "Lsprepost")  520x191+0+0  +0+0
        2 children:
        0x1000b27 (has no name): ()  485x93+11+40  +11+40
        0x1000b1f (has no name): ()  1x1+-1+-1  +-1+-1
     0x10005f9 (has no name): ()  1x1+-1+-1  +-1+-1
     0x1000003 "lsprepost": ("lsprepost" "Lsprepost")  200x200+0+0  +0+0
        1 child:
        0x1000004 (has no name): ()  1x1+-1+-1  +-1+-1
     0x1000001 "lsprepost": ("lsprepost" "Lsprepost")  10x10+10+10  +10+10
        1 child:
        0x1000002 (has no name): ()  1x1+-1+-1  +9+9
     0xe00001 "Xpra Audio record": ("Xpra-Audio-record" "Xpra-Audio-record")  10x10+10+10  +10+10
     0x400026 (has no name): ()  1x1+-1+-1  +-1+-1
     0x400022 "Xpra-SystemTray": ()  1x1+0+0  +0+0
        1 child:
        0x400023 (has no name): ()  1x1+-1+-1  +-1+-1
     0x400020 "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001f "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001e "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001b "Xpra-WorldWindow": ("xpra" "Xpra")  3840x1080+0+0  +0+0
        1 child:
        0x40001c (has no name): ()  1x1+-1+-1  +-1+-1
     0x40001a "Xpra": ()  1x1+0+0  +0+0
     0x400003 "Xpra-ManagerSelection": ()  10x10+-100+-100  +-100+-100
     0x400001 "xpra": ("xpra" "Xpra")  10x10+10+10  +10+10
        1 child:
        0x400002 (has no name): ()  1x1+-1+-1  +9+9
     0x200001 "xpra": ("xpra" "Xpra")  10x10+10+10  +10+10
        1 child:
        0x200002 (has no name): ()  1x1+-1+-1  +9+9

Thu, 31 Jan 2019 13:19:25 GMT - Antoine Martin: owner, status, type changed

I'll assume that xpra sees the same values as xwininfo.

Wed, 13 Feb 2019 11:41:55 GMT - Antoine Martin: attachment set

example app with an embedded opengl subwindow

Wed, 13 Feb 2019 15:46:33 GMT - Antoine Martin: status, description changed; resolution set

Using the example code from Getting Python, GTK and GL to play nice, we can get a test window with a 640x480 opengl view embedded in it. xwininfo sees:

     0x64cafe (has no name): ()  660x635+-10+26  +-10+26
        1 child:
        0x5600003 "ApplicationMainWindowDemo": ("gl_gtk_app.py" "Gl_gtk_app.py")  640x580+10+45  +0+71
           2 children:
           0x5600021 (has no name): ()  640x480+0+75  +0+146
           0x5600004 (has no name): ()  1x1+-1+-1  +-1+70

Done in r21651. The children windows will now also be visible in xpra info | grep children=. Using the test application attached, I see:

windows.1.children=((16777249, 0, 75, 640, 480, 0, 24),)

Which is exactly where the opengl area is, and it is correctly identified as a video area.

It was correctly identified without this code, but the new code gives it a score boost which will help other cases where the existing heuristics might not have spotted it, and it should also help when the subwindow is moved or resized.

@nathan_lstc: video region should work better out of the box for applications with an opengl sub-window.

Wed, 13 Feb 2019 19:03:58 GMT - Nathan Hallquist:

I started an older version of my 3D software (that doesn't talk to dbus) and set the XPRA_CONTENT_TYPE by hand with xprop.

xprop -format _XPRA_CONTENT_TYPE 8s -set _XPRA_CONTENT_TYPE video

When I did that it launches right into h264, which is desired. Then it locks and then it goes for a short time and then it locks and then it goes until I stop rotating. The screen-locking seem to coincide with sburegion recalculations in the log. I've attached a log compress,regiondetect. I will defer to you on whether to reopen this ticket or not. I have the qualitative impression that it is better. I can downgrade and test an older revision, and I will do that if you request it (but I'd rather not).

Wed, 13 Feb 2019 19:04:27 GMT - Nathan Hallquist: attachment set

Thu, 14 Feb 2019 04:12:15 GMT - Antoine Martin: status changed; resolution deleted

Thu, 14 Feb 2019 05:25:55 GMT - Antoine Martin: status changed; resolution set

The pauses that you see are caused by the nvenc re-initializing, as we saw in #2022 the first frame can take a long time to encode:

10:55:50,997 compress: 728.0ms for 1598x892  pixels at    3,29   for wid=2     using      h264 with ratio   1.4%  ( 5574KB to    80KB), sequence    86, client_options={u'speed': 94, u'frame': 0, u'pts': 0, 'csc': 'YUV444P', u'quality': 75}
10:55:51,782 compress: 450.8ms for 1598x892  pixels at    3,29   for wid=2     using      h264 with ratio   0.8%  ( 5574KB to    42KB), sequence    90, client_options={u'speed': 94, u'frame': 0, u'pts': 0, 'csc': 'YUV444P', u'quality': 79}

It starts with x264 and no video region (encoding the whole window as video), which is expected as per r21057 - #2044: we don't want to use nvenc until we are certain that the video stream is here to stay. Then it switches to nvenc (first pause), processes just 4 video frames and then it detects the video region and has to re-initialize nvenc (second pause).

I am closing this ticket again because problem is specific to nvenc and belongs in #2048.

Fri, 15 Feb 2019 15:30:34 GMT - Antoine Martin:

r21670 makes the sub-window region boost configurable using an env var: XPRA_SUBWINDOW_REGION_BOOST (defaults to 20).

Sun, 17 Feb 2019 13:41:21 GMT - Antoine Martin:

r21671 fixes python3 compatibility and more.

Sat, 23 Jan 2021 05:42:33 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/2113