System proxy: create_unix_domain_socket failed for ...

[goekce]

I use Xpra system proxy and login using pam.

• Archlinux, xpra-svn AUR package, xpra v4.1-r27685
• Firefox 81.0.2
• pam-linux: 1.4.0
• I only augmented the systemd file with:

[Service]
Environment=TCP_AUTH=pam
Environment=AUTH=pam
Environment=DEBUG=auth,proxy,util,x11

So the resulting proxy command is:

/usr/bin/python /usr/bin/xpra proxy :14500 --daemon=no --tcp-auth=pam --ssl-cert=/etc/xpra/ssl-cert.pem --ssl=on --bind=none --auth=pam --socket-dirs=/run/xpra --socket-permissions=666 --log-dir=/var/log --pidfile=/run/xpra.pid --debug=auth,proxy,util,x11

What I did:

• open ​http://server_ip:14500 on Firefox
• server forwards me back to connect.html
• I use local account credentials and press on Login.

• (if I use start desktop: xfce4-session, then the behavior is similar)

• server says "websocket connection established"
• times out with:

You were disconnected for the following reason:

Did not receive hello before timeout reached, not an Xpra server?

Here is journalctl excerpt after successful pam auth.

xpra[1738]: 2020-10-18 09:29:12,497 get_server_state: connect(/run/user/1000/xpra/server_hostname-0)=[Errno 111] Connection refused (timeout=5)
xpra[1738]: 2020-10-18 09:29:12,497 ECONNREFUSED
xpra[1738]: 2020-10-18 09:29:12,498 socket_details: '/run/user/1000/xpra/server_hostname-0' state does not match (UNKNOWN vs LIVE)
xpra[1738]: 2020-10-18 09:29:12,598 socket_details(1000, 'LIVE', ':0') sockdir=/run/user/1000/xpra, sockdirs=['/run/user/1000/xpra', '/run/xpra'], testing=['/run/user/1000/xpra', '/run/xpra']
xpra[1738]: 2020-10-18 09:29:13,028 add_process(<subprocess.Popen object at 0x7f5d623edb20>, server-:0, xpra start, True, True) pid=1860
xpra[1738]: 2020-10-18 09:29:13,029 add_dead_process(ProcInfo({'pid': 1860, 'name': 'server-:0', 'command': 'xpra start', 'ignore': True, 'forget': True, 'callback': None, 'process': <subprocess.Popen object at 0x7f>
xpra[1738]: 2020-10-18 09:29:13,029 start_new_session(..) pid=1860, socket_path=/run/user/1000/xpra/server_hostname-0, display=:0,
xpra[1738]: 2020-10-18 09:29:13,030 start_new_session('u', '..', 1000, 1000, {}, [])=(<subprocess.Popen object at 0x7f5d623edb20>, '/run/user/1000/xpra/server_hostname-0', ':0')
xpra[1738]: 2020-10-18 09:29:13,030 connect=True, connect_test_request=None
xpra[1738]: 2020-10-18 09:29:13,030 start_proxy(WebSocket(ws socket: ::ffff:server_ip, 14500, 0, 0 <- ::ffff:client_ip, 48186, 0, 0), {..}, None) using server display at: :0
xpra[1738]: 2020-10-18 09:29:13,043 display description(:0) = {'display_name': ':0', 'type': 'unix-domain', 'local': True, 'display': ':0', 'socket_dirs': ['/run/user/$UID/xpra', '/run/xpra'], 'uid': 1000, 'gid': 10>
xpra[1738]: 2020-10-18 09:29:13,044 socket_details(0, 'LIVE', ':0') sockdir=/run/user/1000/xpra, sockdirs=['/run/user/1000/xpra', '/run/xpra'], testing=['/run/user/1000/xpra', '/run/xpra']
xpra[1738]: 2020-10-18 09:29:13,045 server connection=unix-domain socket:  <- /run/user/1000/xpra/server_hostname-0
xpra[1738]: 2020-10-18 09:29:13,046 start_proxy_process()
xpra[1738]: 2020-10-18 09:29:13,047 sigchld(17, <frame at 0x55fb90f41230, file '/usr/lib/python3.8/site-packages/xpra/platform/displayfd.py', line 62, code read_displayfd>)
xpra[1738]: 2020-10-18 09:29:13,047 poll() procinfo list: [ProcInfo({'pid': 1860, 'name': 'server-:0', 'command': 'xpra start', 'ignore': True, 'forget': True, 'callback': None, 'process': <subprocess.Popen object a>
xpra[1738]: 2020-10-18 09:29:13,047 reap() calling os.waitpid(-1, 'WNOHANG')
xpra[1738]: 2020-10-18 09:29:13,047 sigchld(17, <frame at 0x55fb90c27400, file '/usr/lib/python3.8/subprocess.py', line 1867, code _communicate>)
xpra[1738]: 2020-10-18 09:29:13,048 poll() procinfo list: [ProcInfo({'pid': 1860, 'name': 'server-:0', 'command': 'xpra start', 'ignore': True, 'forget': True, 'callback': None, 'process': <subprocess.Popen object a>
xpra[1738]: 2020-10-18 09:29:13,048 reap() calling os.waitpid(-1, 'WNOHANG')
xpra[1738]: 2020-10-18 09:29:13,081 start_proxy_process(..) client connection=ws socket: ::ffff:server_ip, 14500, 0, 0 <- ::ffff:client_ip, 48186, 0, 0 
xpra[1738]: 2020-10-18 09:29:13,081 start_proxy_process(..) client state={'max_packet_size': 1048576, 'large_packets': [b'hello', b'window-metadata', b'sound-data', b'notify_show', b'setting-change', b'shell-reply',>
xpra[1738]: 2020-10-18 09:29:13,503 ProxyInstance({}, [], {'display_name': ':0', 'type': 'unix-domain', 'local': True, 'display': ':0', 'socket_dirs': ['/run/user/$UID/xpra', '/run/xpra'], 'uid': 1000, 'gid': 1000, >
xpra[1738]: 2020-10-18 09:29:13,504 ProxyProcess(1000, 1000, {}, {}, '/run/xpra', [], ws socket: ::ffff:server_ip, 14500, 0, 0 <- ::ffff:client_ip, 48186, 0, 0, {'display_name': ':0', 'type': 'unix-domain', 'l>
xpra[1738]: 2020-10-18 09:29:13,504 starting proxy instance pid 1738 from pid=1738
xpra[1738]: 2020-10-18 09:29:13,509 ProxyInstanceProcess started
xpra[1738]: 2020-10-18 09:29:13,509 add_process(<multiprocessing.popen_fork.Popen object at 0x7f5d61af98b0>, xpra-proxy-:0, xpra-proxy-instance, True, True) pid=2017
xpra[1738]: 2020-10-18 09:29:13,509 handover complete: closing connection from proxy server
xpra[1738]: 2020-10-18 09:29:13,510 sending socket-handover-complete
xpra[2017]: 2020-10-18 09:29:13,510 started proxy instance pid 2017 
xpra[2017]: 2020-10-18 09:29:13,511  for client ws socket: ::ffff:server_ip, 14500, 0, 0 <- ::ffff:client_ip, 48186, 0, 0 
xpra[2017]: 2020-10-18 09:29:13,511  and server unix-domain socket:  <- /run/user/1000/xpra/server_hostname-0
xpra[2017]: 2020-10-18 09:29:13,512 ProxyProcessProcess.run() pid=2017, uid=0, gid=0
xpra[2017]: 2020-10-18 09:29:13,513 setproctitle not installed: No module named 'setproctitle'
xpra[2017]: 2020-10-18 09:29:13,514 new uid=1000, gid=1000
xpra[2017]: 2020-10-18 09:29:13,514 registered signal handler <bound method ProxyInstanceProcess.signal_quit of proxy instance pid 2017>
xpra[2017]: 2020-10-18 09:29:13,515 waiting for server message on <multiprocessing.queues.Queue object at 0x7f5d623ff4c0>
xpra[2017]: 2020-10-18 09:29:13,516 DotXpra(/run/xpra, ['/run/xpra'] - 1000:1000 - u).socket_path(:proxy-2017)=/run/xpra/server_hostname-proxy-2017
xpra[2017]: 2020-10-18 09:29:13,516 received proxy server message: socket-handover-complete
xpra[2017]: 2020-10-18 09:29:13,516 setting sockets to blocking mode: (ws socket: ::ffff:server_ip, 14500, 0, 0 <- ::ffff:client_ip, 48186, 0, 0, unix-domain socket:  <- /run/user/1000/xpra/server_hostname-0)
xpra[2017]: 2020-10-18 09:29:13,517 set_blocking(ws socket: ::ffff:server_ip, 14500, 0, 0 <- ::ffff:client_ip, 48186, 0, 0)
xpra[2017]: 2020-10-18 09:29:13,517 calling <socket.socket fd=11, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::ffff:server_ip', 14500, 0, 0), raddr=('::ffff:client_ip', 48186,>
xpra[2017]: 2020-10-18 09:29:13,517 create_control_socket: socket path='/run/xpra/server_hostname-proxy-2017', uid=1000, gid=1000, state=DEAD
xpra[2017]: 2020-10-18 09:29:13,517 set_blocking(unix-domain socket:  <- /run/user/1000/xpra/server_hostname-0)
xpra[2017]: 2020-10-18 09:29:13,518 create_unix_domain_socket failed for '/run/xpra/server_hostname-proxy-2017'
xpra[2017]: Traceback (most recent call last):
xpra[2017]:   File "/usr/lib/python3.8/site-packages/xpra/server/proxy/proxy_instance_process.py", line 218, in create_control_socket
xpra[2017]:     sock, self.control_socket_cleanup = create_unix_domain_socket(sockpath, 0o600)
xpra[2017]:   File "/usr/lib/python3.8/site-packages/xpra/net/socket_util.py", line 52, in create_unix_domain_socket
xpra[2017]:     listener.bind(sockpath)
xpra[2017]: PermissionError: [Errno 13] Permission denied
xpra[2017]: 2020-10-18 09:29:13,518 calling <socket.socket fd=13, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/run/user/1000/xpra/server_hostname-0>.setblocking(1)
xpra[2017]: 2020-10-18 09:29:13,520 Error: failed to setup control socket '/run/xpra/server_hostname-proxy-2017':
xpra[2017]: 2020-10-18 09:29:13,520 waiting for server message on <multiprocessing.queues.Queue object at 0x7f5d623ff4c0>
xpra[2017]: 2020-10-18 09:29:13,520  [Errno 13] Permission denied

When I look afterwards for running xpra processes, I see as many processes as the number of login trials. An example forked command:

/usr/bin/python /usr/bin/xpra start --attach=no --env=XPRA_PROXY_START_UUID=d091040f77234c7283333f5cb958e223 --daemon=yes --systemd-run=no --uid=1000 --gid=1000 --displayfd=14

[goekce]

Is it possible to edit the ticket description?

[Antoine Martin]

Please provide more details as per wiki/ReportingBugs.
At the very least: OS, version and command lines.
And since this is a permissions / authentication issue - more details about this setup.

Is it possible to edit the ticket description?

It is now.

[goekce]

Replying to Antoine Martin:

At the very least: OS, version and command lines.
And since this is a permissions / authentication issue - more details about this setup.

I updated the description. How can I help further?

[Antoine Martin]

There's probably something we can do to fail more gracefully, but your config is borked:

... --pidfile t.pem ..  --socket-dirs=/run/xpra.pid

You seem to have mixed up some options.

[goekce]

Sorry journalctl normally does not wrap lines, so it was a copy paste mistake. I updated the description Antoine.

There's probably something we can do to fail more gracefully,

You mean more verbosity could help?

[goekce]

I found a solution on:​https://gitlab.uwaterloo.ca/salt/state/blob/f0106049a872ac6ee0abb9d671b6e6a85b6471cb/math/service/jump/01/xpra/init.sls. Apparently one sysadmin had similar permission problems. Here is an excerpt from this page:

## sadly user in the group xpra does not cut it..
## I remember seeing some change in kernel that unix sockets are more picky about things now.
## perhaps related?
{{sls}} /run/xpra:
    file.directory:
        - name: /run/xpra
        - user: root
        - group: xpra
        - dir_mode: '0777'


## after reboot.. debian resets dir with different mode
## so workaround that with a cron that runs after reboot
{{sls}} force mode 0777 on /run/xpra:
    cron.present:
        - name: 'chmod 0777 /run/xpra'
        - user: root
        - special: '@reboot'

On my machine /run/xpra has the permissions root:root 755. After I changed the permissions to 775 and the group of /run/xpra to xpra-users and added the user to the xpra-users group the problem was solved. But I am not very satisfied with this workaround.

Moreover the error message for users who are not in xpra-users group do not get a meaningful error message.

Do you have a better suggestion?

[Antoine Martin]

On my machine /run/xpra has the permissions root:root 755

That's wrong, we explicitly request 0775: browser/xpra/trunk/src/tmpfiles.d/xpra.conf

Still, when create_unix_domain_socket fails, we should terminate the proxy instance. Re-opening.

[goekce]

Oh thank you, I had indeed removed the /run/xpra directory during debugging. When xpra service starts again, it creates the dir with 755,root:root. I think systemd-tmp.files.d-setup only kicks in during boot, and not when a service is restarted.

The command

sudo /usr/bin/systemd-tmpfiles --create --remove

restores /run/xpra permissions.

Then the admin must pay attention that the users should be in xpra group. Or alternatively another group can be defined in /etc/tmpfiles.d like

d /run/xpra 0775 - xpra-users

Still, when create_unix_domain_socket fails, we should terminate the proxy instance. Re-opening.

Currently I have no idea how I can help to solve this.

[Antoine Martin]

When xpra service starts again, it creates the dir with 755,root:root

I think we just honour the umask - will check.

Currently I have no idea how I can help to solve this.

You can't, I'll take care of it.

[Antoine Martin]

I think we just honour the umask - will check.

So, we don't actually create this directory ourselves - systemd does when it creates the socket for the socket activation - and it looks like it just honours the default umask.
As of r27766, we will try to create the directory ourselves if it is missing or fix its permissions if those are wrong.

But, as of r27767 we should no longer have write access to /run, so this won't actually work.. But at least you'll get a clearer error message.

Still, when create_unix_domain_socket fails, we should terminate the proxy instance.

Improvements in:

• r27770 + r27772 : fixes the leak
• r27774 + r27775 : try harder to fix the directory ownership and permissions

Both the directory mode and group can be changed, the defaults are:

XPRA_SOCKET_DIR_MODE=755
XPRA_SOCKET_GROUP=xpra

[Antoine Martin]

See ticket:2964#comment:4, XPRA_SOCKET_GROUP no longer exists, use:

• XPRA_GROUP to choose the group, applies to unix domain sockets and the socket directories
• XPRA_SOCKET_DIR_GROUP to override the group on the socket directories only

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/2907