macos moduleset updates for 3.0

[Antoine Martin]

See #1985 for 2.5, and pull from ​​https://github.com/GNOME/gtk-osx
Log URL: log/xpra/trunk/osx

Since v3 will be supported for years, we have to try to stay close to the upstream stable moduleset.

[Antoine Martin]

After merging from upstream in r23490 + r23491 + r23492 + r23493, jhbuild did the usual thing: fail with an utterly unhelpful message, very much like #1392.
Why are they so keen on swallowing exceptions everywhere?
Adding print statements in various places, it turns out that the real cause is this:

failed to parse https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules: unknown module type meson

[Antoine Martin]

The meson issue is because we have to re-bootstrap gtk-osx... More info here: ​https://gitlab.gnome.org/GNOME/gtk-osx/tree/master.

Also, to workaround the https issues with the old version of wget found in macos, just install a newer one (TODO: move this to moduleset):

curl -O https://ftp.gnu.org/gnu/wget/wget-1.20.3.tar.gz
tar -zxvf wget-1.20.3.tar.gz
cd wget-1.20.3
./configure --with-ssl=openssl --prefix=$JHBUILD_PREFIX
make
make install

Then we also have to set CA_CERTIFICATE.

[Antoine Martin]

To make this work (PITA):

• had to pull a new version of jhbuild by hand and install it - why?
• had to install meson by hand - why?

That's OK for the python3 / gtk3 variant, but for the python2 / gtk2 environment, there is no python3 there and meson requires it...
During setup the gtk-osx-setup.sh script said something about installing a pipenv with python3, somewhere? who knows?
Do I have to restore the old modules and make a GTK2 URL for it?

[Antoine Martin]

Then later, glib complained that ​https://ninja-build.org/ was not installed.. Yay, yet another build system to deal with. (looking at the setup script, it should have been pulled automatically, oh well)
And this one requires a weird incantation to install (and still complained about something):

python3 ./configure.py --bootstrap
cp ninja $JHBUILD_PREFIX/bin

[Antoine Martin]

Other things that broke: JHBUILD_PREFIX/lib/charset.alias went MIA.

Starting again from scratch on a clean new 10.11 "El Capitan" VM - first for GTK2:

• sh gtk-osx-setup.sh - mentions that we can activate the virtualenv using pipenv shell.
• alias jhbuild="PATH=.new_local/bin:$PATH jhbuild"
• jhbuild bootstrap-gtk-osx
• stick our jhbuild-customrc config in ~/.config (is that even documented anywhere? the gtk-osx home page still refers to the old location..)
• disable cloudflare for xpra.org because jhbuild fails miserably otherwise.. (SSL errors, 403 errors, etc)
• run jhbuild build and watch numerous modules fail starting with gtk..(then pygtk, gtk-mac-integration-python, gtkglext, pygtkglext, etc)

But also:

• python-lzo
• rencode
• pyobjc requires xcode and not just the command line tools... so first install that
• gtkglext directory was empty - the hack to apply the patches before building no longer works because the path has changed.. yay, fixing it by hand fails later during compilation: libtool: compile: unable to infer tagged configuration
• gtk-mac-bundler fails
• bomutils
• serf

The gtk error is caused by the newer pango version missing some macros, simply re-adding them to the pango.h fixes things, but then pygtk still fails later.
So r23506 (+r23507 fixup) downgrades pango back to version 1.42.4 - doesn't gtk-osx still support gtk2?
r23508 also reverts the harfbuzz + freetype-no-harfbuzz changes needed by the older version of pango but keeps the newer freetype version.
(it builds as long as you skip the tests - TODO: make a patch or use a different branch for GTK2 / GTK3?)

Next, try GTK3...

[Antoine Martin]

More GTK2 difficulties:

• r23509: rencode build fix
• r23511: prevents import errors causing the py2app step to fail
• had to patch gtk-mac-bundler to avoid pango errors - why!?
• r23512 new library dependencies that aren't picked up automatically..

[Antoine Martin]

GTK3: on the same system, with the same version of xcode.
Running the gtk-osx setup script and trying to bootstrap fails to link xz properly, like it is using the targeting the wrong SDK version. WTH!?

[Antoine Martin]

More updates:

• r23514 python (+r23520 fix rencode URL)
• r23515 bootstrap
• r23516 network
• r23517 gstreamer: faad2
• r23518 osx python: six (+r23519 fixup)

[Antoine Martin]

So, the build errors are now also occurring with the first user - at least this part makes sense.
They are caused by xcode 8.x: ​homebrew: python3 failed to build on 10.11.6 with Xcode 8: There's an "issue" (Apple considers it a feature) with the new Xcode, which happens as a consequence of Apple retaining the single SDK structure rather than having one SDK in Xcode for 10.11 and another for 10.12.
More info ​here.
Patching dozens of projects for some Apple-only SDK version breakage would be crazy, so I'm now downgrading to xcode 7 instead.

[Antoine Martin]

GTK3:

• bootstrap errors on itstool, something about python needing libxml2 - similar to ​configure: error: Python module libxml2 is needed to run this package but now for python3 - using itstool 2.0.2 fixes the problem (uses python2)
• pango: need to skip tests - create patch?
• gtk-mac-bundler path is wrong during build - untar from source archive and it proceeds
• numpy build error
• python-lzo needs lzo

Still TODO:

• itstool must be built with PYTHON=/path/to/python2 ./configure ...
• fix theming?
• charset.alias?
• gtk-mac-bundler pango patch?

[Antoine Martin]

GTK builds fail with:

$ python3 -c "from gi.repository import Gtk"

** (process:786): WARNING **: 18:33:15.061: Failed to load shared library 'libgdk_pixbuf-2.0.0.dylib' referenced by the typelib: dlopen(libgdk_pixbuf-2.0.0.dylib, 9): image not found
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/importer.py", line 145, in load_module
    importlib.import_module('gi.repository.' + dep.split("-")[0])
  File "/Users/gtk3/gtk/inst/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/importer.py", line 145, in load_module
    importlib.import_module('gi.repository.' + dep.split("-")[0])
  File "/Users/gtk3/gtk/inst/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/importer.py", line 146, in load_module
    dynamic_module = load_overrides(introspection_module)
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/overrides/__init__.py", line 118, in load_overrides
    override_mod = importlib.import_module(override_package_name)
  File "/Users/gtk3/gtk/inst/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/overrides/GdkPixbuf.py", line 32, in <module>
    class Pixbuf(GdkPixbuf.Pixbuf):
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/overrides/__init__.py", line 195, in override
    assert g_type != TYPE_NONE
AssertionError

The typelib paths are strange:

strings $PREFIX/lib/girepository-1.0/*.typelib | grep dylib
libatk-1.0.0.dylib
libgirepository-1.0.1.dylib
/Users/gtk3/gtk/inst/lib/libgobject-2.0.0.dylib,/Users/gtk3/gtk/inst/lib/libglib-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgmodule-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgobject-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgdk-3.0.dylib
libgdk_pixbuf-2.0.0.dylib
libgdk_pixbuf-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgio-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstreamer-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstallocators-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstapp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstaudio-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstbase-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstcheck-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstcontroller-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstgl-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstinsertbin-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstmpegts-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstnet-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstpbutils-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstplayer-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstrtp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstrtsp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstsdp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgsttag-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstvideo-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstwebrtc-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgtk-3.0.dylib,/Users/gtk3/gtk/inst/lib/libgdk-3.0.dylib
/Users/gtk3/gtk/inst/lib/libgtkmacintegration-gtk3.2.dylib
/Users/gtk3/gtk/inst/lib/libpango-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libpango-1.0.0.dylib,/Users/gtk3/gtk/inst/lib/libpangocairo-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/librsvg-2.2.dylib
libcairo-gobject.2.dylib

Adding the library path works around this issue - and hits another one.. (libjpeg):

DYLD_LIBRARY_PATH=$PREFIX/lib python3 -c "from gi.repository import Gtk"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/__init__.py", line 42, in <module>
    from . import _gi
ImportError: dlopen(/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/_gi.cpython-37m-darwin.so, 2): Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Users/gtk3/gtk/inst/lib/libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO

Reproducible with just:

DYLD_LIBRARY_PATH=/Users/gtk3/gtk/inst/lib python3 -c "import gi"

Rebuilding gobject-introspection and pygobject3 does not help.
That's because the ImageIO framework needs the old system libjpeg, not the one we force with DYLD_LIBRARY_PATH..
So we need a way to get gdk-pixbuf to load the correct dylib without using DYLD_LIBRARY_PATH.
Is it a coincidence that gdk-pixbuf is one of the modules that has switched to the meson build system?

[Antoine Martin]

Some pointers:

• ​Crash at startup on OSX "Failed to load shared library"
• ​Incorrect shared-library value in GIRs built on macOS for libraries that use @rpath in install_name
• ​gobject-introspection-1.52.1: cairo-1.0.typelib contains non-absolute path
• ​gobject-introspection: Fix generation of cairo typelib

This does allow us to import gtk:

DYLD_FALLBACK_LIBRARY_PATH=$PREFIX/lib python3 -c "from gi.repository import Gtk"

But using the same env var does not fix running the tests!

Options:

• build without running the unit tests with python3 - no go, still fails during py2app..
• revert the merge that switched to meson?
• don't use turbo-jpeg? (may not help)

Then a new problem during packaging:

ValueError: '/Users/gtk3/gtk/inst/lib/libpython3.7.dylib' does not exist

And indeed, it does not.
We have a libpython3.7m.dylib instead because python was built with --with-pymalloc. Why doesn't py2app figure it out?
Solved by:

cd $PREFIX/lib/                  
ln -sf libpython3.7m.dylib libpython3.7.dylib

[Antoine Martin]

Reverting gdk-pixbuf to autotools in r23523 (+r23524 fixup) fixes that problem.
But then it's atk's turn:

$ python3 -c "from gi.repository import Gtk"

** (-c:48674): WARNING **: 19:58:30.103: Failed to load shared library 'libatk-1.0.0.dylib' referenced by the typelib: dlopen(libatk-1.0.0.dylib, 9): image not found

So r23525 reverts that one to autotools.

[Antoine Martin]

New problem after that: gtk-mac-integration needs rebuilding, it depends on gtk which did get rebuilt. jhbuild should have figured it out...

Moving on, then the unit tests fail again, but later:

FAIL: test_all_codecs_found (__main__.TestDecoders)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gtk3/Xpra/trunk/src/unittests/unit/codecs/codecs_selftest_test.py", line 45, in test_all_codecs_found
    selftest(True)
  File "xpra/codecs/vpx/encoder.pyx", line 739, in xpra.codecs.vpx.encoder.selftest
AssertionError: <module 'xpra.codecs.vpx.encoder' from '/Users/gtk3/gtk/inst/lib/python3.7/site-packages/xpra/codecs/vpx/encoder.cpython-37m-darwin.so'> is limited to 512x512 for vp8 and not 8192x4096

Then running the test by hand produces another strange error:

AssertionError: <module 'xpra.codecs.enc_x264.encoder' from \
    '/Users/gtk3/gtk/inst/lib/python3.7/site-packages/xpra/codecs/enc_x264/encoder.cpython-37m-darwin.so'> is limited to 512x512 and not 8192x4096

That's because of the missing numpy, which is used to generate the test pixel buffers.

[Antoine Martin]

Fixing numpy: revert to v1.16.4 in r23526.

[Antoine Martin]

More GTK3 fixes:

• r23527 gdk-pixbuf packaging error (missing matching pkg-config entry in a GTK3 environment, so hardcoded..)
• r23528 + r23529: don't include anything related to "pygtk"
• r23530: add missing Tango theme

And with these changes, I can finally make both GTK2 and GTK3 builds again!

[Antoine Martin]

And... that's obviously not the end of it.
GTK2 builds have a new problem: the gi bindings are missing, and trying to force enable them in pygobject with --enable-introspection does not work:

  CC     _gi_la-pygi-foreign-gvariant.lo
pygi-info.c:165:14: error: use of undeclared identifier 'GI_INFO_TYPE_ERROR_DOMAIN'
        case GI_INFO_TYPE_ERROR_DOMAIN:
             ^
  CC     _gi_la-pygi-struct.lo
pygi-info.c:135:13: warning: enumeration value 'GI_INFO_TYPE_INVALID_0' not handled in switch [-Wswitch]
    switch (info_type)
            ^
  CC     _gi_la-pygi-argument.lo
pygi-info.c:484:22: error: use of undeclared identifier 'GI_INFO_TYPE_ERROR_DOMAIN'
                case GI_INFO_TYPE_ERROR_DOMAIN:
                     ^
pygi-info.c:448:21: warning: enumeration value 'GI_INFO_TYPE_INVALID_0' not handled in switch [-Wswitch]
            switch (info_type) {
                    ^
  CC     _gi_la-pygi-type.lo
pygi-info.c:863:26: error: use of undeclared identifier 'GI_INFO_TYPE_ERROR_DOMAIN'
                    case GI_INFO_TYPE_ERROR_DOMAIN:
                         ^
pygi-info.c:835:25: warning: enumeration value 'GI_INFO_TYPE_INVALID_0' not handled in switch [-Wswitch]
                switch (info_type) {
                        ^
  CC     _gi_la-pygi-boxed.lo
3 warnings and 3 errors generated.

Apparently, that's because the versions are incompatible: ​pygobject make error.

Looks like the one we want is actually pygobject3.
It builds OK, even against python2, but does not run:

 python2 -c "import gi"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "gi/__init__.py", line 23, in <module>
    from ._gi import _API, Repository
ImportError: No module named _gi

[Antoine Martin]

Ignore the second half of comment:17, which was run from a directory containing a "gi" folder, causing this spurious error.

The new builds do include the gi bindings, but pygtk needed rebuilding (a different version of atk got built at some point?)
Same for gtk-mac-integration-python

[Antoine Martin]

@smo: when you get a chance, you should be able to build clean GTK2-Python2 and GTK3-Python3 environments with trunk.
The only issues you will likely encounter:

• we still have to symlink python to python3 by hand early as jhbuild doesn't handle that
• scons needs python2 to build (so temporarily switch back)
• pango tests fail (needs patch to Makefile to just skip them)
• one or two modules don't build, when you drop to a shell their directory is empty... untar it by hand and continue
• for building: ln -sf libpython3.7m.dylib libpython3.7.dylib (see comment:12)
• itstool issue, see comment:10
• gtk-mac-bundler needs a patch to ignore the magic pango macros it is trying to substitute

[Antoine Martin]

Latest .config/jhbuildrc-custom I'm using for gtk3 builds:

_gtk_osx_use_jhbuild_python = True

setup_sdk(target="10.11", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"

modules = ["meta-osx-xpra-deps"]

moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk3.modules"

os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"

[Antoine Martin]

New error I just hit with flac and the libogg 1.3.4 update from r23655:

error: unknown type name 'uint16_t'
...

Updating flac to 1.3.3 (r23708) did not help.
Fixed by adding #include <stdint.h> near the top of $PREFIX/inst/include/ogg/ogg.h.

[Antoine Martin]

Updates:

• r23751 orc 0.4.30 - reverted in r23752 (requires meson)
• r23776 curl 7.66
• r23787 openssl 1.1.1d
• r23791 python-lz4 2.2.1

[Antoine Martin]

From comment:11 :

Is it a coincidence that gdk-pixbuf is one of the modules that has switched to the meson build system?

No, it's not: ​Need to explicitly export DYLD_FALLBACK_LIBRARY_PATH : It mostly has to do with meson and rpaths, ... I just got GitHub up to date, so if that's where you got it pull and bundle again

Will try again for 4.0 : #2385

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/2383