See wiki/ClientRendering and #350 (which is probably needed first)
In order to bring feature parity with the regular gtk2 pixmap backing, we would need:
GL_MAX_RECTANGLE_TEXTURE_SIZE_ARB
as this won't work.. problem is that we may resize at any time, so maybe we should just not use OpenGL
when there is any risk that the windows will be too big: whenever the total display size is bigger than the texture size. Another solution would be to dynamically switch to PixmapBacking
when needed (as there is little GL specific code in GLClientWindow
anyway)
This will make the code more manageable (fewer special cases), should make the overall experience better (ie: we can then use RGB modes with mmap and gl, x264 can encode to/from rgb without doing csc server side) and will give the opengl more of a chance to be used.
scaling completed in r3906
The problem with video RGB painting is that the data coming out of x264 is in planar mode ("GBRP
") so we cannot re-use _do_paint_rgb24
as is.
Can we update the fbo one plane at a time? Maybe using a new texture for each plane?
Or is our only option to first convert to plain RGB
and re-use the _do_paint_rgb24
code?
The client-side conversion is probably almost as costly as doing YUV444
to RGB
csc server-side before compression, but since we are CPU bound server-side, we are much better off shifting this burden to the client. (effectively lowering the overall latency)
Theoretically we can paint planar RGB. It requires a separate shader and some more support code, however.
Planar RGB done in r3928 + r3927 + r3926 + r3925 + r3923 + r3922 + r3921 + r3920
Looks to me like the new buffer management code (#350) is making PyOpenGL
unhappy (100% reproducible):
OpenGL paint error: ("No array-type handler for type \ <type 'buffer'> (value: <read-only buffer ptr 0x7f394011b910, size 171904 ) registered", \ <OpenGL.GL.images.ImageInputConverter object at 0x2277390>) Traceback (most recent call last): File "xpra/client/gl/gl_window_backing.py", line 376, in gl_paint_planar self.update_planar_textures(x, y, enc_width, enc_height, img, pixel_format, scaling=(enc_width!=width or enc_height!=height)) File "xpra/client/gl/gl_window_backing.py", line 431, in update_planar_textures glTexSubImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, x, y, width/div_w, height/div_h, GL_LUMINANCE, GL_UNSIGNED_BYTE, pixel_data) File "OpenGL/latebind.py", line 45, in __call__ return self._finalCall( *args, **named ) File "OpenGL/wrapper.py", line 781, in wrapperCall pyArgs = tuple( calculate_pyArgs( args )) File "OpenGL/wrapper.py", line 354, in calculate_pyArgs yield converter(args[index], self, args) File "OpenGL/GL/images.py", line 433, in __call__ return arrayType.asArray( arg ) File "OpenGL/arrays/arraydatatype.py", line 141, in asArray return cls.getHandler(value).asArray( value, typeCode or cls.typeConstant ) File "OpenGL/arrays/arraydatatype.py", line 52, in __call__ typ, repr(value)[:50] TypeError: ("No array-type handler for type <type 'buffer'> \ (value: <read-only buffer ptr 0x7f394011b910, size 171904 ) registered", \ <OpenGL.GL.images.ImageInputConverter object at 0x2277390>)
(This particular trace occurred with GBRP
when using quality=100
)
Solutions (most of which assume that the problem is to do with read-only memory - TBC):
PyBuffer_FromMemory
Damn, read-write makes no difference, so maybe PyOpenGL
requires a new style buffer. PITA
This works around the issue by copying to a new string object before passing the data to gl:
--- src/xpra/client/gl/gl_window_backing.py (revision 3977) +++ src/xpra/client/gl/gl_window_backing.py (working copy) @@ -426,7 +426,7 @@ glActiveTexture(texture) glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textures[index]) glPixelStorei(GL_UNPACK_ROW_LENGTH, rowstrides[index]) - pixel_data = img_data[index] + pixel_data = img_data[index][:] debug("texture %s: div=%s, rowstride=%s, %sx%s, data=%s bytes", index, divs[index], rowstrides[index], width/div_w, height/div_h, len(pixel_data)) glTexSubImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, x, y, width/div_w, height/div_h, GL_LUMINANCE, GL_UNSIGNED_BYTE, pixel_data) if index == 1:
r3985 applies the change from comment:10, just so we can move on and use the code, but this will need a better solution eventually.
r4017 uses a string wrapper rather than a buffer wrapper, which makes pyopengl happy and allows us to revert the extra buffer copy added in r3985.
I've left swscale alone, since we don't use it with opengl, but it may need the same treatment if we ever do csc with opengl (for whatever reason).
Items left to do:
GL_MAX_RECTANGLE_TEXTURE_SIZE_ARB
should be exposed to the client class so it can avoid gl windows if the screen size is too big (>4k is not uncommon these days)
r4017 was "wrong": the string wrapper makes a copy... which is what we tried to avoid, so r4048 reverts that and just makes the buffer copy from the decoding thread instead of the UI thread (and only for opengl)... so we still need to find a better way to make the buffer work with zero copy and pyopengl... maybe The new-style Py_buffer struct
Nice-to-have but not essential, re-scheduling.
r4480 made the pixel buffer read/write, so we no longer have to copy the pixels before uploading them with glTexSubImage2D
(see comment:9), removed copying in r4702
working patch for posix clients (win32 broken)
r4880 + r4881 + r4883 adds transparency support for GL windows, r4879 (server side), backported in r4894 so that older servers don't end up sending BGRA pixels without telling the client the data is in BGRA and not RGBA... (as is normally the case when we don't specify)
This change is already very useful for mmap/rgb modes where we now handle 32-bit pixel data in native BGR(A) format, so the server no longer needs to do any byteswapping and/or losing the alpha channel, it will also be very useful with vp9/h265 which both support alpha channels.
Buffer improvements task is now in its own ticket: #465
Last remaining item for this ticket is the maximum GL texture size, which is made more difficult by #263: we cannot just set a maximum window size as this also changes the behaviour of the UI..
r4882 + r4883 improves texture size checking and ensures we have a fallback in case GL window setup fails. We already had a check to disable GL if the total screen space is bigger than the maximum texture supported. Ideally, we would also deal with failures during window resize, as one can resize a window and make it bigger than the screen size... but this will do for now (modern graphics card support 16k x 16k texture sizes which is more than we need).
One remaining niggle is the fact that I cannot get OpenGL
to use the premultiplied alpha despite following the blending docs... (see r4902)
afarr: please test combinations I do not have access to (as of r4977, one needs to use XPRA_ALPHA=1 xpra attach ...
to try to enable alpha channel on osx and win32 where it is normally disabled by default):
To make sure that there are no regressions - the performance improvements are too small to be measurable under normal usage.
The video encodings (vpx
and h264
) should be unchanged, rgb
, png
and webp
should support window transparency (xpra.test_apps.transparent_colors can be used to test transparency easily) on OpenGL
windows (verify window type is GL in session info).
This belongs in the 0.11 milestone, new ticket added for resizing flickering for 0.12: #478
Oooops, wrong milestone update
Testing with win client 0.11.0 r5144 (fedora 19 0.11.0 r5144): with rgb
and png
the transparent colors look lovely. With webp
the squares are the right colors but streaked with black (not lovely).
Testing with osx client 0.11.0 r5144 produces the same results: rgb
and png
work as expected and webp
doesn't support the transparencies.
The webp issue is not GL related (which is what this ticket is about), moved to #487
Closing.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/385