Xpra: Ticket #1575: python3 packaging for macos

python3 tracker ticket: #1568.

TODO:

etc..



Wed, 12 Jul 2017 12:25:38 GMT - Antoine Martin: status changed

This may fix issues like #761 and #404, worth a try.

Does the GTK3 macos port support opengl though? (#1569)


Mon, 06 Nov 2017 09:00:43 GMT - Antoine Martin: owner, status changed

@smo: any updates?


Sun, 12 Nov 2017 07:18:31 GMT - Antoine Martin: priority changed

Needed for window transparency: #1570


Sun, 26 Nov 2017 13:33:46 GMT - Antoine Martin: milestone changed


Sun, 26 Nov 2017 13:35:53 GMT - Antoine Martin:

See also #1708: 2.3 moduleset updates


Sat, 30 Dec 2017 10:07:45 GMT - Antoine Martin: priority changed

@smo: please put anything you have already done in this ticket, then I can take a look.


Wed, 03 Jan 2018 03:20:07 GMT - Smo:

I tried to build the up to date moduleset but had a problem with pygobject. Here is my contact to the mailing list which didn't really go anywhere.

https://mail.gnome.org/archives/gtk-osx-devel-list/2017-September/msg00000.html


Wed, 03 Jan 2018 15:45:00 GMT - Antoine Martin: owner, status changed

Using a clean user account and following the build instructions from https://wiki.gnome.org/Projects/GTK+/OSX/Building:

wget --no-check-certificate \
    "https://git.gnome.org/browse/gtk-osx/plain/gtk-osx-build-setup.sh"
sh ./gtk-osx-build-setup.sh
PATH=/Users/gtk3/.local/bin:$PATH
jhbuild bootstrap

Got a few certificate errors with: xz, pkg-config, flex, expat, etc (download without checking certs and continue). Then it fails to build python3 with another cert error:

jhbuild buildone python3
jhbuild buildone: could not download http://git.gnome.org/browse/gtk-osx/plain/modulesets-stable/gtk-osx.modules: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

This time fixed by downloading the cacert.pem from here:

https://curl.haxx.se/docs/caextract.html and then telling jhbuild to use it:

CA_CERT_FILE=$HOME/cacert.pem jhbuild build python3

Make python3 the default (as per docs - though the upstream instructions look wrong for the second symlink):

jhbuild shell
cd $JHBUILD_PREFIX/bin
ln -s python3 python
ln -s python3-config python-config
jhbuild build openssl

(upstream notes apply: interrupt the build and run it by hand - we should upstream our openssl 1.1 build commands)

jhbuid build meta-gtk-osx-bootstrap meta-gtk-osx-gtk3

Got a minor failure building glib:

xzcat -d "/Users/gtk3/gtk/source/pkgs/glib-2.52.2.tar.xz" | tar xf -
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-gint64-long-long.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-gint64-long-long.patch"
patching file configure.ac
Hunk #1 succeeded at 3002 (offset 48 lines).
Hunk #2 succeeded at 3114 (offset 48 lines).
Hunk #3 succeeded at 3266 (offset 49 lines).
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch"
patching file gio/gosxappinfo.c
patching file gio/gosxcontenttype.c
Hunk #1 succeeded at 52 (offset -6 lines).
patching file gio/tests/contenttype.c
Hunk #1 succeeded at 323 with fuzz 1 (offset -38 lines).
Hunk #2 FAILED at 359.
1 out of 2 hunks FAILED -- saving rejects to file gio/tests/contenttype.c.rej
*** Error during phase checkout of glib: ########## Error running patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch" *** [14/39]

Applied the one line from the reject file by hand and continued.

Everything built correctly after that.

@smo: whatever you did (which was not recorded anywhere AFAICT) must have been different?


Continuing from there:

Got one failure we should fix on lame:

gzip -dc "/Users/gtk3/gtk/source/pkgs/lame-3.99.5.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [8/23]
patch -p0 < "/Users/gtk3/.cache/jhbuild/lame-channels.patch"
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/libmp3lame/set_get.c
|+++ b/libmp3lame/set_get.c
--------------------------
File to patch:
Skip this patch? [y]
Skipping patch.
patch: **** malformed patch at line 10:

Next we have to build python and switch back to python3 afterwards so we can build the python deps against python3:

jhbuild buildone python
jhbuild shell
cd $JHBUILD_PREFIX/bin
rm python python-config
ln -s python3 python
ln -s python3-config python-config

Build the rest:

jhbuild build meta-osx-xpra-python-deps

lz4 had a problem (will fix). Then python3 had a problem finding ssl, it just needed to be rebuilt now that we brought in a newer openssl lib version:

jhbuild buildone python3 -f

At that point, everything is built OK, AFAICT.

Still TODO:


Thu, 04 Jan 2018 07:06:24 GMT - Antoine Martin:

@smo: why are we so far behind https://git.gnome.org/browse/gtk-osx/tree/? Even gtk2 is outdated here. (only our gstreamer and network modulesets are slightly ahead) Is there anything there that we should not merge and why?

I've merged most of the changes:

Also added latest gdb by hand:

wget https://ftp.gnu.org/gnu/gdb/gdb-8.0.1.tar.gz
tar -zxvf gdb-8.0.1.tar.gz
cd gdb-8.0.1
./configure --prefix=${JHBUILD_PREFIX}
make && make install
sudo chgrp procmod ${JHBUILD_PREFIX}/bin/gdb
sudo chmod g+s ${JHBUILD_PREFIX}/bin/gdb

But... still needs signing to be used: codesign gdb.


Thu, 04 Jan 2018 11:43:50 GMT - Antoine Martin:

More work on this:

Still TODO:

Apart from that, it looks usable. Beta builds posted here with the "Python3" filename: https://xpra.org/beta/osx/.


Fri, 05 Jan 2018 18:50:44 GMT - Antoine Martin:

Notes after rebuilding again from scratch to verify:

Saves having to temporarily switch back to python2 to build it.

The python2-only builds started hitting the issue from comment:7, and now the mixed python2-python3 builds do too. Sigh.

One more weird issue started happening: the "py2app" step does something that loads gdk and tries to access the display server and crashes when it fails, means that the builds now have to be executed from a GUI shell. PITA.


So at this point, the python3 jhbuild env is almost straightforward (bar python3 vs python issues) with this file:

cat > .jhbuildrc-custom  <<EOF
_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "openssl", "python3", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core", "meta-gtk-osx-gtk3",
"libpng",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-mac-integration-python", "meta-osx-python3-xpra-deps"]
moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
EOF

Sat, 06 Jan 2018 12:51:27 GMT - Antoine Martin: owner, status changed

Lots more updates (in large part because testing jhbuild is hard, so I commit then fix the numerous mistakes one at a time). I've also sent some questions and pull requests upstream to try to reduce our diff:

Changesets:

With all these changes:

Then just: jhbuild build. The only difference for the "GTK2" moduleset is the moduleset URL:

moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk2.modules"

Then when building the packages, we have to do:

PYTHON=python3 ./make-all.sh

(or PYTHON=python2 for GTK2 builds)

@smo: does that work for you?


Mon, 08 Jan 2018 15:42:12 GMT - Antoine Martin:

Note: the gi bingings were missing from the python2 builds (meaning no sound), to re-instate them I had to apply this patch: pygobject: remove references to deprecated GI_INFO_TYPE_ERROR_DOMAIN. (basically, just remove all references to GI_INFO_TYPE_ERROR_DOMAIN then rebuild with introspection enabled) There might be a better way of doing this.


Wed, 10 Jan 2018 16:47:44 GMT - Smo:

gtk-engines fails when building for python3/gtk3 is this needed?

checking for GTK... no
configure: error: GTK+-2.12 is required to compile gtk-engines
*** Error during phase configure of gtk-engines: ########## Error running ./configure --prefix /Users/xpragtk3/gtk/inst    *** [47/129]

Wed, 10 Jan 2018 16:50:20 GMT - Antoine Martin:

gtk-engines fails when building for python3/gtk3 is this needed?

It isn't referenced anywhere in our modulesets. Can you get some info on it? jhbuild info gtk-engines


Tue, 16 Jan 2018 18:02:24 GMT - Smo:

Name: gtk-engines
Module Set: gtk-osx-themes
Type: autogen
Install version: not installed
Install date: not installed
URL: http://ftp.gnome.org/pub/GNOME/sources/gtk-engines/2.20/gtk-engines-2.20.2.tar.bz2
Version: 2.20.2
Tree-ID: 2.20.2-d41d8cd98f00b204e9800998ecf8427e
Sourcedir: /Users/xpragtk3/gtk/source/gtk-engines-2.20.2
Requires: automake, libtool, make
Required by: meta-gtk-osx-themes
After: meta-gtk-osx-core

Wed, 17 Jan 2018 04:25:22 GMT - Antoine Martin:

meta-gtk-osx-themes is found here: https://github.com/GNOME/gtk-osx/blob/master/modulesets-stable/gtk-osx-themes.modules, but isn't referenced anywhere else in the modulesets. There is a meta-gtk-osx-gtk3-core-themes, so I've switched to that in r18020. @smo: does that fix things? Do we need to add icon-naming-utils?


Thu, 17 May 2018 19:41:02 GMT - Smo:

Started from scratch again today ran into some issues with libxml2. It would always just build the python2 variant and then would also fail with

ld: -stack_size option can only be used when linking a main executable

We can see the configure line here

./configure --prefix /Users/xpragtk3/gtk/inst --libdir="$JHBUILD_LIBDIR" --with-python  --with-python-install-dir=/Users/xpragtk3/gtk/inst/lib/python2.7/site-packages

I'm not sure where the --with-python-install-dir comes from it doesn't appear in the moduleset?

If you run configure again and change it to the correct python install dir it seems to install minus having to stop the build to use the same line as comment:11

python -c "import libxml2; print(libxml2)"
<module 'libxml2' from '/Users/xpragtk3/gtk/inst/lib/python3.6/site-packages/libxml2.py'>

seems to load properly now


Thu, 17 May 2018 20:01:30 GMT - Smo:

itstool 2.0.2 doesn't work with python3 itstool 2.0.4 works with python3 but segfaults when used building gtk-doc itstool from git seems to work just fine had to install this manually


Thu, 17 May 2018 22:55:50 GMT - Smo:

seems to be some problems with building lame still as well

** Checking out lame *** [105/135]
curl --continue-at - -L http://winswitch.org/src/lame-3.100.tar.gz -o /Users/xpragtk3/gtk/source/pkgs/lame-3.100.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3983  100  3983    0     0  11362      0 --:--:-- --:--:-- --:--:-- 11380
*** Error during phase checkout of lame: file hash is incorrect (expected ddfe36cab873794038ae2c1210557ad34857a4b6bdc515785d1da9e175b1da1e, got 30641cf26a0d4dc26ca6b97e719c52b4c72d49014a09cd28309a9d989a0408a2) *** [105/135]

Hash is wrong but the url is actually invalid for the download. Should this be switched or was there a good reason for having it on winswitch.org?

*** Checking out lame *** [105/135]
gzip -dc "/Users/xpragtk3/gtk/source/pkgs/lame-3.100.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [105/135]
patch -p1 < "/Users/xpragtk3/.cache/jhbuild/lame-channels.patch"
patching file libmp3lame/set_get.c
Hunk #1 FAILED at 91.
1 out of 1 hunk FAILED -- saving rejects to file libmp3lame/set_get.c.rej
*** Error during phase checkout of lame: ########## Error running patch -p1 < "/Users/xpragtk3/.cache/jhbuild/lame-channels.patch" *** [105/135]

After getting the correct tarball this patch still fails.

If I ignore the patch failing and try to build I get build errors like these

Undefined symbols for architecture x86_64:
  "_lame_init_old", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libmp3lame.la] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of lame: ########## Error running make -j 3  *** [105/135]

Thu, 17 May 2018 22:57:12 GMT - Smo:

gst-plugins-base has issue building the examples

libtool: link: unable to infer tagged configuration
libtool:   error: specify a tag with '--tag'

Don't build the examples with --disable-examples configure flag and everything builds fine.


Thu, 17 May 2018 23:28:45 GMT - Smo:

scons doesn't build under python3

*** Building scons *** [131/135]
python setup.py build
scons: *** SCons does not run under Python version 3.6.4.
Python 3 and above are not yet supported.
*** Error during phase build of scons: ########## Error running python setup.py build *** [131/135]

need this to build serf and subversion


Thu, 17 May 2018 23:29:19 GMT - Smo:

Bunch of warnings I noticed as well when you start the build

W: meta-osx-xpra-python3-deps has a dependency on unknown "gst-python" module
W: meta-osx-xpra-python3-deps has a dependency on unknown "python3-pyasn1" module
W: meta-osx-xpra-python3-deps has a dependency on unknown "python3-pyasn1" module
W: python3-pyasn1-modules has a dependency on unknown "python3-pyasn1" module
W: python3-pyasn1-modules has a dependency on unknown "python3-pyasn1" module
W: python3-ldap has a dependency on unknown "python3-pyasn1" module
W: python3-ldap has a dependency on unknown "python3-pyasn1" module

Mon, 21 May 2018 04:50:24 GMT - Antoine Martin:

Started from scratch again today ran into some issues with libxml2. It would always just build the python2 variant and then would also fail with

Weird, I didn't hit this one.

itstool from git seems to work just fine had to install this manually

Again, I didn't hit this one. (I'm building on 10.10) We'll just wait for the next release and update itstool in the moduleset then.

seems to be some problems with building lame still as well

Yep. It's an upstream bug:

gst-plugins-base has issue building the examples

Fixed in r19356.

I hit a more fundamental issue rebuilding gstreamer:

  GEN      Gst-1.0.gir
(..)
ImportError: dlopen(/Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so, 2): Symbol not found: _PyInt_FromLong
  Referenced from: /Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so
  Expected in: flat namespace
 in /Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so

No idea when the broke, or why. Had to rebuild gobject-introspection which caused dozens of other modules to get rebuilt.

Then something else was not quite right - looks like some headers aren't installed by gst-plugings-base, gst-libav fails with:

/Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/audio.h:36:10: fatal error: 'gst/audio/gstaudioaggregator.h' file not found
#include <gst/audio/gstaudioaggregator.h>
         ^
1 error generated.

Fixed by:

cp ${JHBUILD_SOURCE}/gst-plugins-base-1.14.1/gst-libs/gst/audio/gstaudioaggregator.h ${JHBUILD_PREFIX}/include/gstreamer-1.0/gst/audio/

Then after that:

In file included from /Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/audio.h:36:
/Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/gstaudioaggregator.h:27:10: fatal error: 'gst/base/gstaggregator.h' file not found
#include <gst/base/gstaggregator.h>
         ^
1 error generated.

Fixed by:

cp ${JHBUILD_SOURCE}/gstreamer-1.14.1/libs/gst/base/gstaggregator.h ${JHBUILD_PREFIX}/include/gstreamer-1.0/gst/base/

Bunch of warnings I noticed as well when you start the build pyasn1 ...

r19355 fixes that.


Thu, 24 May 2018 19:12:42 GMT - Smo:

They seem to have a fix for the lame issue over at homebrew site

https://github.com/Homebrew/homebrew-core/blob/master/Formula/lame.rb

    # Fix undefined symbol error _lame_init_old
    # https://sourceforge.net/p/lame/mailman/message/36081038/
    inreplace "include/libmp3lame.sym", "lame_init_old\n", ""

Just remove the offending line from include/libmp3lame.sym and all is well. I'll add a patch.


Thu, 24 May 2018 21:51:47 GMT - Smo:

r19418 adds the patch to the repo r19419 adds it to the moduleset

lame now builds without error


Fri, 01 Jun 2018 11:45:22 GMT - Antoine Martin: status changed; resolution set


Sat, 23 Jan 2021 05:28:08 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1575