python3

[Antoine Martin]

Tracker ticket for everything related to python3.

Split from #640 / #90.
See also:

• packaging: #1253 split packages, #1258 "python2" package naming
• #1041 python3 for gstreamer on win32 (no longer used since #678)
• #1386 python3 for gstreamer sound process (other platforms)
• #1717 python3 gtk3 client
• #1569 python3 opengl client
• #1574 python3 packaging for win32
• #1575 python3 for macos
• #1589 GTK3 clipboard support
• #853 python3 server

[Antoine Martin]

Subcommands with python3:

• attach: needs opengl: #1569
• detach: OK
• screenshot: fixed in r16267
• info: OK
• control: OK
• print: OK
• version: fixed in r16269
• showconfig: OK
• list-mdns: fixed in r16274
• mdns-gui: fixed in r16275
• shadow: subcommand now hidden in r16277 (needs work)

Server bits are moved to their own ticket: #1571.

[Antoine Martin]

nvenc fixes and dependency updates: r16399 + r16400 + r16401 (see ticket:1550#comment:14)

[Antoine Martin]

Part of r16607 doesn't work on python3:

Warning: unable to override socket object
 SSL peek support will not be available
 module 'socket' has no attribute '_socketobject'

See #1504 for details.

[Antoine Martin]

Lots of updates in r17015 + r17017. The shadow server is now usable with python3, only needs a few minor fixes:

• encoding options aren't set properly (csc warnings, no video encoders)
• clipboard is disabled
• ssl socket peek wrapper
• errors during cleanup caused by py3k needless breakage ie: RuntimeError: dictionary changed size during iteration in cancel_damage : for sequence in self.statistics.encoding_pending.keys():

[Antoine Martin]

More updates and fixes:

• r17025: clipboard
• r17026 + r17029: keyboard
• r17027: shadow server image encoding
• r17028: fix ssl, gdk Rectangle
• r17091: rfb and websockets

Python3 shadow servers are now totally usable and fast. Not much left to fix here:

• cursor icons
• SIGINT (general problem with GTK3)

TODO:

• the client still needs graphs in session info, maybe we should allow export to CSV or include more context information in the graph (see ticket:1705#comment:1)
• websockify needs patching (removing ForkingMixIn)
• server core needs a different workaround for non-blocking sockets
• problem with mouse cursors?
• XSETTINGS / root props, see r6384 (needs more GTK glue)
• transparency may work without opengl?

[Antoine Martin]

r17098 is wrong, the bug is still present in all GTK3 versions! ​Ctrl+C does not exit gtk app - absolutely amazing that such a fundamental issue has not been fixed in so many years.
It does mean that we may be scheduling timers too often, which is why we exit the main loop and catch the signal. Worth looking into.

[Antoine Martin]

Lots more fixes: r17498, r17507, r17509.

• bug report tool needs a fix for the clipboard copy (probably easy).
• the systray sort of works with gnome-shell if you install ​topicons plus, but not with GTK3... where the geometry comes up as 1x1, and even forcing it does not help

[Antoine Martin]

Found a blocker bug with "de" keyboard layout and win32 clients: I can get a client crash just by pressing the first key to the right of the left shift key!

ERROR:../../pygobject-3.26.1/gi/pygi-argument.c:1004:_pygi_argument_to_object: code should not be reached

Why give a helpful warning when you can just crash the whole application, thanks GTK! sigh.

[Antoine Martin]

Updates:

• r17582: fixup rencode for python 3.6 import warnings
• r17579 + r17581: lots of deprecation warnings fixed

Some import warnings were completely hidden behind a generic message until I used this badly documented gem:

PYTHONWARNINGS='error::ImportWarning' python3 /usr/bin/xpra attach

The remaining warnings are:

• pyopengl: adding from __future__ import absolute_import to the pyx files doesn't work, it breaks somewhere else trying to cimport withoud pxd files (how is that supposed to work!?)
• ​Gdk.Screen: most functions are deprecated and we're supposed to Use per-monitor information - could be done
• ​Gtk.Menu.set_title is deprecated - no replacement... probably not a big issue
• ​Gtk.ImageMenuItem: GTK developers are hellbent on trying to force everyone to not use icons in menu items - the most bone headed move yet (anyone who has ever had to use a computer setup to use a language they are unable to read can testify here)
• ​StatusIcon: everything is deprecated because gnome decided to break the systray one more time (a tie with the icon thing for being the worst decision)

[Antoine Martin]

The signal handler has finally been fixed, it only took GTK ~7 years to fix a basic unix feature. Considering that other platforms (win32 and macos) are barely supported... wow.
The changesets needed to pygobject are:

• ​Make Python OS signal handlers run when an event loop is idling
• ​Install a default SIGINT handler for functions which start an event loop

If these changes apply cleanly to the pygobject version in MSYS2, let's patch it.

[Ray Donnelly]

Both those patches to GTK are from a very active MSYS2 developer so hopefully there'll be official packages sometime soon.

[Antoine Martin]

Hit a concurrency error with python3, fixed in r19101. Problem is that there are likely many more like this one because they have changed the semantics of values(), keys() and items() on dictionaries. Major pain.
Wherever we fix this by using a temporary list, we unnecessarily penalise python2. A version specific wrapper function would be better, but would still cost an extra function call.

[Antoine Martin]

rpmbuild moans about those:

*** WARNING: mangling shebang in /usr/lib/cups/backend/xpraforwarder from #!/usr/bin/env python to #!/usr/bin/python2. This will become an ERROR, fix it manually!
*** WARNING: mangling shebang in /usr/libexec/xpra/auth_dialog from #!/usr/bin/env python to #!/usr/bin/python2. This will become an ERROR, fix it manually!
*** WARNING: mangling shebang in /usr/libexec/xpra/xdg-open from #!/usr/bin/env python to #!/usr/bin/python2. This will become an ERROR, fix it manually!

[Antoine Martin]

Lots of GTK3 issues trying to fix deprecation warnings shown with PYTHONWARNINGS=all, some fixes in r19697, r19698, r19699, r19700, r19701.

​Gtk.StatusIcon: the whole class is deprecated, ie: there is no direct replacement for this function. Oh great, good thing we're only using this on X11

Also found a crasher bug: #1886.

[Antoine Martin]

Regarding comment:19, those shebangs get mangled by rpmbuild (see ​fedora packaging committe: mangling shebangs and #1891), using the totally undocumented ​__brp_mangle_shebangs_exclude_from (as per ​zfs commit : Exclude python scripts from RPM shebang check) prevents that: r19812.

[Antoine Martin]

websockify 0.8 modified to run on python 3.7

[Antoine Martin]

See also: GTK4 #1977

A good summary of how bad the situation is with python3 strings vs bytes: ​The Python Unicode Mess

[Antoine Martin]

See also #2121.
SSL socket peek support is now properly fixed: r21877.

Still TODO:

• client session info graphs

[Antoine Martin]

Everything seems to work fine, including the proxy server (#2185).

The few remaining bits are things like #2195, which have their own ticket.

[MayeulC]

Replying to Antoine Martin:

See also: GTK4 #1977

A good summary of how bad the situation is with python3 strings vs bytes: ​The Python Unicode Mess

Another good read is ​https://drewdevault.com/2017/01/13/The-problem-with-Python-3.html

The situation really isn't as bad as it is being painted there. Thank you for porting xpra to python3, hopefully I can now remove (again) python2 from my (wayland, sigh) system once the change trickles to the Arch packages :)

[Antoine Martin]

Another good read is ​​https://drewdevault.com/2017/01/13/The-problem-with-Python-3.html

Awful. The guy just doesn't get it.
Please count the hundreds of pointless changesets we've had to make for python3 (still ongoing) and tell me again why this is "better".