OSX PKG signing

[Antoine Martin]

Follow up form #641. (see also #840)

Links:

• ​Signing Your Code Manually
• ​Technical Note TN2206: macOS Code Signing In Depth

Once the certificate have magically appeared in Xcode and the keychain (..), we can sign the Xpra.app bundle, the PKG and DMG files:

codesign --deep --force --verify --verbose --sign "Developer ID Application: Antoine Martin (XXXXXXXXXX)" Xpra.app
codesign --deep --force --verify --verbose --sign "Developer ID Application: Antoine Martin (XXXXXXXXXX)" Xpra.dmg
productsign --sign "Developer ID Installer: Antoine Martin (XXXXXXXXXX)" Xpra.pkg Xpra-signed.pkg

The signature can be verified with:

codesign --verify --verbose=2 Xpra.app
spctl --assess --verbose=4 --type=execute Xpra.app
spctl --assess --verbose=4 --type=execute Xpra.dmg
spctl --assess --verbose=4 --type=install Xpra-signed.pkg

But this fails for signing "Xpra.app" if I keep the Xpra_NoDock.app nested inside the Xpra.app, it complains:

image/Xpra.app: invalid Info.plist (plist or signature have been modified)
In subcomponent: /Users/osx/Xpra-svn/trunk/osx/image/Xpra.app/Contents/Xpra_NoDock.app

Removing the nested app fixes this, but there must be something I can change in the Info.plist (or elsewhere?) to be able to bundle it.

@smo: what am I missing?

[Antoine Martin]

(edit: added verify for PKG - full error message)

[Antoine Martin]

App store submission steps:

• ​Creating an iTunes Connect Record for an App
• ​Uploading Your App to iTunes Connect
• ​Submitting the App to App Review

[Antoine Martin]

Despite the hopelessly unhelpful and misleading error message from the codesign tool (..), I've narrowed it down through trial and error (mostly error):

• removing all symlinks from the subapp passes
• removing Contents/MacOS from the subapp passes
• using the Info.plist from the main app passes

Bringing the Info.plist closer to the main one "fixes" things, done in r14152.

Along the way I found that changing CFBundleExecutable is not possible in the subapp!? It's fortunate that we don't use this executable directly.

Now onto app submission. Let's hope it is as much fun as this, not.

[Antoine Martin]

r14154 + r14155 adds hooks to do the signing as part of the app / dmg / pkg scripts.

So until I move to a newer build host, the process is:

• build the app (currently on a 10.5.8 VM):

  ./make-app.sh
  

• copy the "Xpra.app" over to the machine doing the signing, ie:

  rsync -rplogt --delete "image/Xpra.app" osx@osx1010:/Users/osx/Xpra-svn/trunk/osx/image/
  

• login to the signing VM and unlock the keychain:

  security unlock-keychain -p THEACCOUNTPASSWORD
  

(see ​User interaction is not allowed: trying to sign an OSX app using codesign)

• run the remaining part of the build:

  cd Xpra-svn/trunk/osx
  export CODESIGN_KEYNAME="Antoine Martin"
  ./sign-app.sh && ./make-DMG.sh && ./make-PKG.sh 
  

---

Appstore fun:

• follow steps in comment:2
• install ​ApplicationLoader 3.0
• watch in despair as the tool fails with yet another set of cryptic messages - for which there does not seem to be any reference or documentation anywhere, some have been fixed (ie: r14160 for "ITMS-90242"), most have not:
  • "ITMS-90251" : The product archive is invalid. Javascript may not be used in the installer.
  • "ITMS-90230" : Invalid product archive metadata. Error in keyPath [product-metadata.product-identifier]...
  • "ITMS-90230" : Invalid product archive metadata. Error in keyPath [product-metadata.product-version]...
  • "ITMS-90237" : The product archive package's signature is invalid. Ensure that it is signed with your "3rd Party Mac Developer Installer" certificate
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Path "Applications/Xpra?.app/Contents/Resources/lib/python2.7/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap" in Payload is missing from Bom.
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra?.app" has identifier "Xpra" in PackageInfo? but "org.xpra.Xpra" in Payload
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra?.app" has version CFBundleVersion="1.3.0" in PackageInfo?, but CFBundleShortVersionString="1.0", CFBundleVersion="1.0" in Payload
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra?.app/Contents/Xpra_NoDock.app" in Payload not declared in PackageInfo?
  • "ITMS-90000" : Bad Bundle Executable. You must include a valid CFBundleExecutable key in your bundle's information property list file
  • "ITMS-90000" : Unsupported toolchain. Packages submitted to the App Store must be created either through Xcode, or using the productbuild tool..

[Antoine Martin]

More fixes:

• r14153: fix CFBundleIdentifier
• r14156: add ITSAppUsesNonExemptEncryption=false
• r14157: get version used for generating the DMG and PKG filenames from app and not local environment
• r14158 + r14159 + r14164: what goes in CFBundleSignature??
• r14160: add LSApplicationCategoryType
• r14161: fix build stripping (backported in r14165)
• r14162: remove installation-check script (we don't care about OSX versions older than 10.5), use correct CFBundleIdentifier (as per r14153 above)
• r14163: use a nested sub bundle for the sound "No Dock" sub app, sign with the PKG with the "3rd Party Mac Developer Installer" key (see ​Confusion around Signing Mac App Installer and ​Difference between “Mac Developer” and “3rd Party Mac Developer Application”)

What remains:

• for javascript: ​About Distribution Definition Files: Important: For applications distributed on the App Store, use Xcode to make your installer package. If you need to specify system requirements, use the Pre-install Requirements Property List in the Xcode build settings. For more details, see the productbuild man page. - not sure how to fix this one
• product-metadata issues: ​Distribution XML Reference: product - sadly, adding <product id="org.xpra.Xpra" version="$VERSION" /> to the Distribution file makes things worse, Application Loader then claims not to find the file you click on: The file \U201cXpra-1-1.0-r14164\U201d couldn\U2019t be opened because there is no such file (this error is just garbage)
• CFBundleExecutable could be caused by the lack of signature on "Xpra_Launcher"?

  $ ls -al@ image/Xpra.app/Contents/MacOS/
  total 24
  drwxr-xr-x   5 osx  staff   170 14 Oct 20:00 .
  drwxr-xr-x  10 osx  staff   340 14 Oct 20:03 ..
  -rwxr-xr-x@  1 osx  staff  2752 10 Oct 21:44 PythonExecWrapper
  	com.apple.cs.CodeDirectory	 141 
  	com.apple.cs.CodeRequirements	 180 
  	com.apple.cs.CodeSignature	8527 
  -rwxr-xr-x@  1 osx  staff   399 10 Oct 21:44 Xpra
  	com.apple.cs.CodeDirectory	 128 
  	com.apple.cs.CodeRequirements	 164 
  	com.apple.cs.CodeSignature	8526 
  -rwxr-xr-x   1 osx  staff   292 14 Oct 20:00 Xpra_Launcher
  

---

See also:

• ​Flat Package Format - The missing documentation
• ​Unpacking an OS X pkg file, ​How to unpack and pack pkg file?
• ​Packages packaging freeware
• the "Application Upload" tool can be used from the command line which makes it easier to copy the error messages (even though the output is pretty badly formatted, at least cut&paste is possible..):

  "/Applications//Application Loader.app/Contents/Frameworks/ITunesSoftwareService.framework/Support/altool" -v -f ./image/Xpra-*.pkg -u 'THEAPPLEID' -p 'THEPASSWORD'
  

• ​Packaging a daemon for OS X
• ​Making OS X Installer Packages from Mountain Lion to Lion or Snow leopard
• ​docosx: OSX flat packages
• ​Downloading installer packages from the Mac App Store with AppStoreExtract

[Antoine Martin]

Alternatively using productbuild:

productbuild --component ./image/Xpra.app \
    /Applications \
    ./image/Xpra-1.0.pkg  \
    --sign "3rd Party Mac Developer Installer"

The resulting installer package lacks any icons or text but seems to install OK.
Only one error remains for "Application Loader" upload for this one: ITMS-90259: Bad Bundle Executable...

Or even:

pkgbuild --analyze --root ./image/Xpra.app ./image/XpraComponents.plist
#edit XpraComponents.plist and set {{{BundleIsRelocatable}}} to false
pkgbuild --root ./image/Xpra.app \
    --component-plist ./image/XpraComponents.plist \
    --install-location "/Applications" \
    ./Xpra.pkg
#generate the Distribution.xml file:
productBuild --synthesize --package ./Xpra.pkg ./Distribution.xml
# modify it using the same data as found in our make-PKG.sh script, and:
# change "#base.pkg" to "#Xpra.pkg"
# add the product id and version line
productbuild --distribution ./Distribution.xml --package-path . \
    --sign "3rd Party Mac Developer Installer" \
    ./Xpra-Installer.pkg

Same "bad bundle executable" error, and also missing the 512x512 icons...

---

More links:

• ​OS X: Creating Packages from the Command Line - Tutorial and a Makefile - Part I
• ​How to build a Mac OS X Installer Package from these files?
• ​mac: How to add a License.txt to a pkg built with productbuild using --component option?
• ​Making OS X Installer Packages like a Pro - Xcode Developer ID ready pkg

[Antoine Martin]

I can get a little bit further by:

• replacing the "MacOS" shell scripts with a python binary (temporarily) - scripts should work though, as long as they are more than 28 characters long (this is not a joke: ​http://superuser.com/a/463673/58851)
• deleting the "no dock" sub-app (temporarily)
• deleting the "Helpers" (temporarily)
• signing all "dylib", "so", "MacOS/*" and "Resources/bin" files using:

  codesign --force --verbose --sign "3rd Party Mac Developer Application" --entitlements ./Xpra.entitlements  THEFILE
  

The entitlements file was added in r14166. (it's not clear if the same entitlements should be used for all the files: I doubt the system is capable of isolating the code running in libraries from its entrypoint)

• signing the "image/Xpra.app" after that

---

Then you need to make an upload to itunes... which takes a lot of time and effort. And then you get some more errors via email! (this is some kind of bad joke) Invalid Signature: (..) satisfies its Designated Requirement test-requirement: code failed to satisfy specified code requirement(s) - could they have worded it more ambiguously perhaps?
Before I started signing the "so" / "dylib" / etc, the error was different for those: code object is not signed at all In architecture: i386).
Rinse and repeat until you find the right combination.
But you can't delete versions on there, so you have to bump the CFBundleVersion every time. How convenient.

---

More links:

• ​Application failed codesign verification. What do I do?
• ​How do you codesign framework bundles for the Mac App Store?
• is CODESIGN_ALLOCATE env var needed? ​How to codesign an existing Mac OS X .app file for gatekeeper?

[Antoine Martin]

to make the package non relocatable

[Antoine Martin]

work in progress to generate a PKG compatible with the appstore

[Antoine Martin]

A big part of the difficulty is that it's non-trivial to embed shell scripts... Because shiny. Or something. (and also because it takes way too long to test anything)

With the patch attached, the codesign step shows:

(..)
image/Xpra.app/Contents/Helpers/Bug_Report: replacing existing signature
image/Xpra.app/Contents/Helpers/Bug_Report: signed generic [Bug_Report]
(..)

Note that is uses "generic" and not "signed Mach-O thin (i386)" like it does for all the binaries. And then much much later (..), you get an email (..) saying the build cannot be uploaded:

org.xpra.Xpra.pkg/Payload/Xpra.app: code object is not signed at all
In subcomponent: org.xpra.Xpra.pkg/Payload/Xpra.app/Contents/Helpers/Bug_Report

See:

• ​How to codesign a shell script executable inside a Mac app bundle?
• ​Can you code sign executable files like bash scripts?
• ​Is signing of shell script wrapping applications broken in OS X 10.11? - brittle
• ​AppCheckerLite for checking signatures and entitlements

Maybe we can change the wrappers so that the "MacOS" binary executes the real "Python" executable (renamed if needed) with the script as argument instead. (and move the scripts somewhere else..)

Still TODO:

• fix sound subapp
• postinstall script with symlinks to /usr/bin
• agent

etc.

[Antoine Martin]

A gem in ​http://prod.lists.apple.com/archives/xcode-users/2013/Oct/msg00067.html:
In addition to my entitlements file I needed to create an info.plist file and embed it into the binary by adding the following linker flag:

-sectcreate __TEXT __info_plist SandboxUnixCommand/SandboxUnixCommand-Info.plist

Well, obviously... ahem. Magic TEXT section values!

[Antoine Martin]

updated patch

[Antoine Martin]

Lots more nasty tricks to beat this mess into shape:

• r14167 + r14174: dynamically populate version / revision / buildno during build
• r14168: fix typo
• r14169 + r14171 + r14172 + r14173 + r14176 + r14180: move things aroung to make codesign happy (some of these may not be mandatory - trial and error..)
• r14170: we don't use the webcam yet, remove the entitlement
• r14175: we need this weird entitlement!?
• r14177: fix code stripping (backport in r14182)
• r14178: fix "Bug Report" launch script (backport in r14181)
• r14183: appstore packaging script

[Antoine Martin]

At last a package that both runs and uploads to the appstore:

• r14187 + r14189 + r14192 + r14193: generate an Info.plist for each wrapped utility
• r14188: improve C wrapper script

[Antoine Martin]

appstore view

[Antoine Martin]

as shown on the ​https://itunesconnect.apple.com/ appstore control page:

[Antoine Martin]

Sound sub-app fixes:

• r14223: move binaries to "/Resources/bin"
• r14224 + r14226: run the "sandbox inherit" version of the sound binary
• r14225, r14222: minor related tweaks

[Antoine Martin]

Some path fixes and updates:

• r14266 (backport in r14267)
• r14268
• r14269: generate an SSL cert (but postinstall script is still not used in appstore version..)

Ideally, we would store our preferences in /Library/Preferences, but:

• there is no API to get the path to this directory (sigh), see ​NSSearchPathDirectory
• we are only supposed to use XML plist files in there? (sigh)

Then, we could handle ​Migrating an App to a Sandbox and have the preferences moved for us to the user sandbox.

[Antoine Martin]

I don't think this is going to happen anytime soon. Re-scheduling.

Their complaints after reviewing the app:

• load session file dialog can trigger errors if the folders are not accessible due to sandboxing (ie: the home folder is not), not much we can do here since the file selection dialog is GTK's - we could just remove the load and save options for apple users (simplicity at its best), or use a different file selection tool (native somehow?), or remove the sanbox restrictions? (yuk)
• name of the app doesn't match ("Launcher" vs "Xpra") and "Quit" is "Disconnect Launcher" - no biggie, this could be changed easily enough
• we require sandboxing exceptions:
  • com.apple.coreservices.launchservicesd is needed to for pyobjc "coreservices" - not much we can do about this - maybe try to silence / ignore the errors this causes, or report the pyobjc issue upstream
  • com.apple.systemuiserver.screencapture is needed for using xpra as a server - they unhelpfully suggested: We encourage you to investigate other ways of implementing the desired functionality.
• we use CTFontCopyDefaultCascadeList which is a non-public API, see ​pango coretext: use public function to obtain cascade list if available - looks like rebuilding on 10.8 or newer should fix this: #840

[Antoine Martin]

Moving the appstore packaging to a new ticket: #1366

This purpose of this ticket is now only about signing the PKG.
At least some of the latest PKG builds should be signed... I should make it easier to know which PKG are signed and which ones are not. (requires using a newer VM to sign.. which won't boot right now on my systems.. even with the Intel one..)

I'm not entirely sure what the actual difference is for users when installing signed vs unsigned packages. I've paid the fee for the privilege of having Apple sign the key. Does this make it easier to install the PKG on OSX versions with the more stringent security settings?

[Antoine Martin]

Success. I have fixed my signing 10.10.x VM (it needs vbox 5.0 to boot, fails with 5.1 - fixes for running 5.0 on Fedora 25: ​Error: nothing provides libvpx.so.3()(64bit) on Fedora 25 and ​Runtime/r0drv/linux: 4.8 mod_timer_pinned fix).

Uploaded both DMG and PKG r14467 builds signed with my developer key.
@afarr: do these signed builds help with installation on newer versions of macosx? (you may need to turn the "stricter installation checks" setting back on again)

[alas]

Odd.

I have an r14467 .dmg installed in my usual odd place - deleting and re-installing into the more usual /Applications directory works as usual, and likewise with the r14492 .dmg.

Double clicking on them, however, they fail the "Verifying 'Xpra'" validation test, "because it is from an unidentified developer".

I can't find any sign of the r14467 pkg though, or .dmg, in either the /dists or the /beta directories (so, where did I get the r14467 .dmg I already have?).

Trying to install the r14492 .pkg I get the usual security/unsigned (unknown developer) issue, but I'm not sure you signed that one.

I think you'll need to post another signed .pkg or even .dmg for me to check.

[Antoine Martin]

1.0 should be signed, just released today.
(the other ones disappeared because I needed to free some diskspace)

---

Note to self since this is not obvious, this is how we make the signed DMG / PKG:

• run ./make-app.sh on the build VM
• copy the Xpra.app to the signing VM
• unlock the keys on the singing VM: security unlock-keychain -p THEACCOUNTPASSWORD
• run ./make-DMG.sh && ./make-PKG.sh on the signing VM

[alas]

Checking in xpra.org/beta/osx ... the 1.0 r14492 pkg (​http://xpra.org/beta/osx/Xpra-1.0-r14492.pkg) as well as the xpra.pkg (​http://xpra.org/beta/osx/Xpra.pkg) both, once downloaded, trigger a security warning when trying to install (“Xpra.pkg” can’t be opened because it is from an unidentified developer.).

They indicate that they were posted/updated 11/27.

Handing this back to you... and chuckling that my username seems to have become 'alas' (I feel three breaths closer to Elizabethan-dom ... three breathless breaths, whatever that means).

[Antoine Martin]

1.0 was released, meaning it is no longer in the beta area! Try here instead: ​http://xpra.org/dists/osx/x86/.
(and the files in the beta area will soon disappear altogether to make way for 2.0 builds)

As for 'alas', that's your "name" in your account settings. The trac update now shows the name rather username. You should be able to change it yourself from your account profile - but it does have a nice ring to it, doesn't it?

[Antoine Martin]

1.0.1 is also signed.

[alas]

unknown developer error when installing .pkg

[alas]

Err — alas... I seem to be triggering the same "unknown developer" error with the new 1.0.1 (Xpra-1.0.1-r14723.pkg) that I was mentioning getting with 1.0 in comment:20 (which I also repeated with Xpra-1.0-r14502).

I got the errors with the new 10.12.1 super-shiny mac OS, and then confirmed I'm also getting it with a 10.10.5 significantly-less-shiny mac.

Same message with 1.0 and 1.0.1, on 10.12.1 & 10.10.5.

Apple must have more hoops for you?

[Antoine Martin]

Does it work any better with the beta 2.0 builds?
Those are built and signed on the same system.

[J. Max Mena]

Sadly, the latest 2.1 betas are still complaining about unidentified developers.

[Antoine Martin]

The certificate expired a few months back so I stopped signing the builds, you may want to try a build made around the time of comment:25 instead. (2.0.x)

[J. Max Mena]

Retried with a build from mid March, and still got the error, and that was the oldest 2.0 build available in the betas folder (dists had nothing relevant unless I missed something that obvious..which is possible).

Either way, based on the conversations we've had, I'm going to close this - if we want to revisit OSX signing, we can re-open this, but in the meantime, closing.

(As an aside, I think we need a "crickets" resolved option, but that's just me)

[Antoine Martin]

See also #2015, #2441.

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/1340