2.0 moduleset updates

[Antoine Martin]

Now that we've dropped support for older build systems, we can update the modulesets with upstream changes again:

• r14737: gettext, autoconf (safe for 1.x?)
• r14738: websocket_client (will apply to 1.x)
• r14740: libxml2 (safe for 1.x?), gettext (safe for 1.x), gtk-mac-integration (not safe), glib (not safe for 1.x), libffi (safe?), gobject-introspection (not safe), pango (safe?), gdk-pixbuf (not safe?), gtk+-3.0 (not used), pixman (safe?)

[Antoine Martin]

I encountered some build problems on "yelp-xsl", "pango" and "gdk-pixbuf":

dyld: Library not loaded: /Users/osx/gtk/inst/lib/libintl.8.dylib
  Referenced from: /Users/osx/gtk/inst/bin/xgettext
  Reason: image not found

Which clearly we're not the first ones to hit: ​dyld: Library not loaded: /usr/local/libodep/lib/libintl.8.dylib (and many more hits)

Fixed by rebuilding libcroco. Hit a similar thing later on "gtk+", solved by rebuilding "atk".

Then it gets wedged with gobject-introspection:

*** Configuring gobject-introspection *** [27/111]
(..)
checking for flex... flex
checking lex output file root... configure: error: cannot find output from flex; giving up

Fixed by re-installing flex. ("make install" from the source dir - odd that "jhbuild buildone flex" didn't work here, oh well)

Maybe I should have just ran everything from the top ("jhbuild bootstrap").

@smo: any changes to the list of safe/unsafe candidates for applying to the 1.x branch?

[Antoine Martin]

Building from scratch ("jhbuild bootstrap" then "jhbuild build") works fine, just takes longer.

But trying to build the xpra package then fails with:

calling 'gtk-mac-bundler Xpra.bundle' in /Users/osx/Xpra-svn/trunk/osx
Traceback (most recent call last):
  File "/Users/osx/.local/bin/gtk-mac-bundler", line 6, in <module>
    bundler.main.main(sys.argv[1:])
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/main.py", line 18, in main
    bundler.run()
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/bundler.py", line 611, in run
    self.copy_binaries(self.project.get_binaries())
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/bundler.py", line 207, in copy_binaries
    dest = self.copy_path(path)
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/bundler.py", line 224, in copy_path
    source = self.project.evaluate_path(Path.source)
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/project.py", line 288, in evaluate_path
    path = utils.evaluate_pkgconfig_variables(path)
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/utils.py", line 27, in evaluate_pkgconfig_variables
    raise Exception("pkg-config variable '%s %s' is undefined" % (key, module))
Exception: pkg-config variable 'pango_module_version pango' is undefined
ERROR: gtk-mac-bundler failed

Following the advice from ​Make pango modules related code optional to fix build with pango 1.38, r14746 removes pango from the bundle file.
We still need the patch to be applied to gtk-mac-bundler (it applies cleanly - skipping the examples), this needs to be added to the moduleset.

[Smo]

r14752 adds patch for gtk-mac-bundler without examples
r14753 upgrades to newer version of gtk-mac-bundler and applies new patch also old patch is not necessary anymore

[Antoine Martin]

One of those updates, likely r14753, broke our packaging scripts, r14760 fixes that.

[Smo]

r14782 add patch for pygtkglext
r14785 renames patch
r14786 uses the patch in the moduleset
r14787 enabled dependency for pygobject3 which should build gtk3 properly now.

[Antoine Martin]

Works for me too, closed ticket:840#comment:37.

Can you comment on which libraries are safe to apply to 1.0? (see ticket description).
My guess: gettext and autoconf only? gdk-pixbuf?

[Smo]

Sorry for the delay.

This is what I think

shoud be safe
gettext, autoconf, libxml2, gettext

maybe
pango, pixman, gdk-pixbuf

not safe
gtk-mac-integration, libffi

[Antoine Martin]

• applied the "should be safe" in r14861
• tested then applied the "maybe" list in r14862

Notes:

• jhbuild build is not enough - you may get errors like in comment:1, best to start again from the top: jhbuild bootstrap -f;jhbuild build -f
• rsvg was moaning:

  Making all in .
    CCLD     rsvg-view-3
  ld: library not found for -lgtk-3
  collect2: ld returned 1 exit status
  make[2]: *** [rsvg-view-3] Error 1
  make[1]: *** [all-recursive] Error 1
  make: *** [all] Error 2
  

Fixed by: jhbuild buildone -f gtk+-3.0

• libtheora got stuck on SDL detection (odd - don't care)

[Antoine Martin]

Was wrong about the pixman update, reverted in r14904.

[Antoine Martin]

r14910 also updates the bundler and properly fixes the issues with the new pango version

[Antoine Martin]

For the record: my VM got hosed by a system update.. (wouldn't boot no matter what bootloader or boot CD I tried to use..) so I reverted back to an earlier snapshot.. and jhbuild just fell over itself too many times trying to update + ugprade (libintl.8.dylib and more) so I decided to start again from scratch - a bad idea (tm).

That didn't work until I completely wiped the gtk directory. And even then it failed to build jpeg because nasm was the default system one (too old).
Got a few SSL certificate problems along the way too: cairographics.org, gstreamer.freedesktop.org, etc.. PITA
python-cryptography needs openssl, but somehow that wasn't built by then.. and then I found out that the current openssl version in the moduleset had been removed from the download location we use... so I bumped openssl in r14944. (we need permanent urls for everything instead)

I'll then forced a full rebuild (jhbuild build -f) at the end to ensure that everything is linked against the library version we ship. (ie: our newer openssl for python, etc)

[Antoine Martin]

More in r15055:

• ffmpeg 3.2.4
• cryptography 1.7.2
• python-xxhash 1.0.0

[Antoine Martin]

Lots more updates in r15138 (copying changelog message):

• sshpass 1.06
• opus 1.1.4
• setuptools 34.2.0
• pycparser 2.17
• cffi 1.9.1
• curl 7.53.0
• pyasn1 0.2.2
• py2app 0.12
• numpy 1.12
• macholib 1.8
• modulegraph 0.14
• altgraph 0.13
• ipaddress 1.0.18
• idna 2.2
• enum34 1.1.6
• scons 2.5.1
• subversion 1.9.5

Some of those should be backported sooner rather than later: sshpass, opus - and maybe also numpy.

[Antoine Martin]

Two of those update are problematic:

• ​setuptools-34.0 fails with 'ImportError: No module named packaging.version'
• cffi seems to break cryptography packaging - could be: ​Support for PyInstaller, py2exe, Cx_Freeze and similar tools

I'd rather fix this rather than downgrade again, but can't spend too much time on this either.

[Smo]

r15143 changes setup tools to 33.1.1. There seems to be an issue with 34 which will take some more work to fix.

[Antoine Martin]

• setuptools was downgraded to version 33.1.0 in r15143 to fix the import error from comment:15, but we are actually locked on the much older version 19.2 to fix pyobjc problems, done in r15170
• py2app seems to be a bit borken: it tries to check things (something it didn't do before), and it tries too hard, hitting this bug: ​Importing ctypes.wintypes on Linux gives a ValueError instead of an ImportError.

Then it also seems to get confused about which platform it is running on and blows up some other win32 modules we don't care about. (removing them from the source tree helps, temporarily at least)
Downgrading to 0.11 still shows problems: it fails during its qt5 import recipes (missing 'sip' dependency)... but we don't even use qt, what are you doing there py2app?
So r15165 downgrades py2app from version 0.12 to 0.10, also adds a warning message in the moduleset file.

• cffi: tried a few versions and only versions up to 1.5.2 allow us to package python-cryptography properly, so r15168 downgrades it and adds a note.

---

@smo: just a FYI, things should be packaged OK again. If you had rebuilt python-cryptography with a newer cffi, you may need to run

jhbuild buildone -f python-cryptography

[Smo]

Everything builds fine. Even forced rebuild of cffi and python-cryptography but i'm seeing this unit test fail now.

======================================================================
ERROR: test_perf (__main__.TestCrypto)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/build/xpra-trunk/src/unittests/unit/net/crypto_test.py", line 31, in setUp
    pycryptography_backend.init()
  File "/Users/build/gtk/inst/lib/python2.7/site-packages/xpra/net/pycryptography_backend.py", line 52, in init
    backend = default_backend()
  File "/Users/build/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 71, in default_backend
    _default_backend = MultiBackend(_available_backends())
  File "/Users/build/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/multibackend.py", line 36, in __init__
    "Multibackend cannot be initialized with no backends. If you "
ValueError: Multibackend cannot be initialized with no backends. If you are seeing this error when trying to use default_backend() please try uninstalling and reinstalling cryptography.

----------------------------------------------------------------------
Ran 2 tests in 5.384s

Maybe I need to do something more to get this to work properly again?

[Antoine Martin]

The error in comment:19 is definitely the error that made me downgrade cffi in r15168.
You may need to ensure you don't have an old version laying around (remove all and re-install it). Then you may need to zap the python-cryptography source directory too.
If that still doesn't help, try zapping all the other downgrades (setuptools, etc).

[Smo]

After deleting all directories having to do with setuptools, cffi and python-cryptography I was able to rebuild them then the unittests pass.

Must have been some issue with the downgrades.

I should note I also deleted stuff from

~/gtk/inst/lib/python2.7/site-packages

cffi*, cryptography* and setuptools*

[Smo]

Even on newer versions of osx we still need the package tools to make the pkg.

Making Xpra-x86_64-2.0-r15188.pkg
229163 blocks
4 blocks
mkbom: invalid option -- u
Usage: mkbom [-h] [-s] directory bom
       mkbom [-s] -i filelist bom
PKG Signing skipped (no keyname)
33      ./image/Xpra-x86_64-2.0-r15188.pkg
Done PKG

I built them with this

jhbuild build meta-osx-xpra-pkgtools

[Smo]

Ran into this issue and it appears it is a problem with openssl 1.1. This might take some work to fix. Not sure at the moment. Going to check if we really need this on newer versions of osx.

checking for openssl/evp.h... yes
checking for OpenSSL_add_all_ciphers in -lcrypto... no
configure: error: Cannot build without libcrypto (OpenSSL)
*** Error during phase configure of xar: ########## Error running ./configure --prefix /Users/build/gtk/inst    *** [1/4]


*** module meta-osx-xpra-pkgtools not built due to non buildable xar *** [4/4]

[Smo]

Making Xpra-x86_64-2.0-r15188.pkg
229148 blocks
4 blocks
PKG Signing skipped (no keyname)
34      ./image/Xpra-x86_64-2.0-r15188.pkg
Done PKG

r15189 comments out xar and adds a note

[Antoine Martin]

The pygobject flags patch had gone MIA: I have removed it in the r15149 cleanup because it was not referenced anywhere since r13713, but that was a mistake: we need it to avoid the warnings (see ticket:1444#comment:2).
Re-added in r15191. I wasn't just seeing things then.

[Antoine Martin]

We should do something about the git checkouts: either use a specific git id, or take a snapshot (version it ourselves). ie: libepoxy and others.
And since we also want to be able to apply patches.. maybe snapshots is the way to go?

[Smo]

r15254 adds grep and pcre which is needed for grep

Added these because I needed them to rule out issues with cmake and configuring libyuv. The system grep doesn't support the '-P' option so it errors when running cmake for libyuv.

This didn't fix the issue with libyuv but I figure I should leave them in there in case we need them later.

As for the stuff from git goes if we need to apply any patches with the jhbuild system that doesn't work for us. The only way to do this is with tarball snapshots.

[Smo]

I was updating my builds today and for some reason the dependency for setuptools_scm is being ignored for some reason I see that it was added for lz4 which is why I needed it.

It looks right in xpra.modules but it isn't doing what we want.

[Smo]

Added libyuv now so we can build it within the moduleset and added it to the meta module so it should automatically get built.

details in #1280

[Antoine Martin]

Follow up for 2.1: #1501

Please close unless something still needs doing?