xpra icon
Bug tracker and wiki

Opened 5 years ago

Last modified 3 weeks ago

#173 assigned enhancement

extended input device support

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: minor Milestone: 3.0
Component: core Version: trunk
Keywords: Cc: rektide@…

Description (last modified by Antoine Martin)

think: tablet pcs can be used with the program running on the higher end desktop machine.

Pressure sensitivity, fine-grained scrolling, etc.

Attachments (15)

main_v2.c (4.7 KB) - added by Antoine Martin 7 months ago.
uinput example
xi-events.patch (33.7 KB) - added by Antoine Martin 5 months ago.
handle XI2 events
xi-events-v2.patch (18.7 KB) - added by Antoine Martin 5 months ago.
updated patch for r15792
xi-events-v3.patch (35.5 KB) - added by Antoine Martin 5 months ago.
updated patch with device discovery
xi-events-v4.patch (29.4 KB) - added by Antoine Martin 5 months ago.
updated patch for r15794
xi-events-v5.patch (49.2 KB) - added by Antoine Martin 5 months ago.
updated patch
xi-events-v7.patch (56.4 KB) - added by Antoine Martin 5 months ago.
better approach: register event handlers directly with the XI2 helper class
xi-events-v8.patch (63.9 KB) - added by Antoine Martin 5 months ago.
updated patch: export all device properties
xi-events-v9.patch (68.7 KB) - added by Antoine Martin 4 months ago.
updated patch: server know knows details of the device used for some events
00-ignore-xpra-devices.conf (155 bytes) - added by Antoine Martin 4 months ago.
tell Xorg to always ignore the virtual input devices we create
00-only-xpra-devices.conf (314 bytes) - added by Antoine Martin 4 months ago.
tell Xdummy to add our virtual input devices
61-evdev-xpra.rules (31 bytes) - added by Antoine Martin 4 months ago.
change the udev permissions on input devices
xi-events-v10.patch (79.9 KB) - added by Antoine Martin 4 months ago.
use uinput to inject events
xi-events-v11.patch (79.0 KB) - added by Antoine Martin 4 months ago.
switch to using python-evdev
uinput-server.patch (18.5 KB) - added by Antoine Martin 7 weeks ago.
implement pointer as a uinput device

Download all attachments as: .zip

Change History (41)

comment:1 Changed 5 years ago by Antoine Martin

Milestone: 0.6future
Status: newaccepted

comment:2 Changed 2 years ago by rektide

Cc: rektide@… added

comment:3 Changed 18 months ago by Antoine Martin

Milestone: future0.18

See investigation in ticket:1131#comment:5

Looks like we want uinput.

comment:4 Changed 18 months ago by Antoine Martin

Description: modified (diff)
Summary: xinput forwardingextended input device support

comment:7 Changed 16 months ago by Antoine Martin

And some information from a trackpad which works for smooth scrolling.
From libinput-list-devices:

Device:           DLL0704:01 06CB:76AE Touchpad
Kernel:           /dev/input/event13
Group:            6
Seat:             seat0, default
Size:             101.33x56.67mm
Capabilities:     pointer 
Tap-to-click:     disabled
Tap-and-drag:     enabled
Tap drag lock:    disabled
Left-handed:      disabled
Nat.scrolling:    disabled
Middle emulation: n/a
Calibration:      n/a
Scroll methods:   *two-finger edge 
Click methods:    *button-areas clickfinger 
Disable-w-typing: enabled
Accel profiles:   none
$ xinput list | grep -i touchpad
⎜   ↳ DLL0704:01 06CB:76AE Touchpad           	id=11	[slave  pointer  (2)]
$ xinput list-props 11
Device 'DLL0704:01 06CB:76AE Touchpad':
	Device Enabled (137):	1
	Coordinate Transformation Matrix (139):	1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
	libinput Tapping Enabled (275):	1
	libinput Tapping Enabled Default (276):	0
	libinput Tapping Drag Lock Enabled (277):	0
	libinput Tapping Drag Lock Enabled Default (278):	0
	libinput Accel Speed (279):	0.000000
	libinput Accel Speed Default (280):	0.000000
	libinput Natural Scrolling Enabled (281):	0
	libinput Natural Scrolling Enabled Default (282):	0
	libinput Send Events Modes Available (259):	1, 1
	libinput Send Events Mode Enabled (260):	0, 0
	libinput Send Events Mode Enabled Default (261):	0, 0
	libinput Left Handed Enabled (283):	0
	libinput Left Handed Enabled Default (284):	0
	libinput Scroll Methods Available (285):	1, 1, 0
	libinput Scroll Method Enabled (286):	1, 0, 0
	libinput Scroll Method Enabled Default (287):	1, 0, 0
	libinput Click Methods Available (288):	1, 1
	libinput Click Method Enabled (289):	1, 0
	libinput Click Method Enabled Default (290):	1, 0
	libinput Disable While Typing Enabled (291):	1
	libinput Disable While Typing Enabled Default (292):	1
	Device Node (262):	"/dev/input/event13"
	Device Product ID (263):	1739, 30382
	libinput Drag Lock Buttons (293):	<no items>
	libinput Horizonal Scroll Enabled (264):	1
$ xinput query-state 11
2 classes :
ButtonClass
	button[1]=up
	button[2]=up
	button[3]=up
	button[4]=up
	button[5]=up
	button[6]=up
	button[7]=up
ValuatorClass Mode=Relative Proximity=In
	valuator[0]=467
	valuator[1]=1056
	valuator[2]=-1401
	valuator[3]=2254
$ xinput get-button-map 11
1 2 3 4 5 6 7 
$ xinput get-feedbacks 11
1 feedback class
PtrFeedbackClass id=0
	accelNum is 2
	accelDenom is 1
	threshold is 4

Two finger scrolling as seen with libinput-debug-events:

event13	POINTER_AXIS      +2.73s	vert 6.07* horiz 0.49*
event13	POINTER_AXIS      +2.74s	vert 6.56* horiz 0.49*
event13	POINTER_AXIS      +2.75s	vert 6.89* horiz 0.82*

xev only sees pairs of button events:

ButtonPress event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291002, (106,47), root:(144,149),
    state 0x10, button 5, same_screen YES

ButtonRelease event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291002, (106,47), root:(144,149),
    state 0x1010, button 5, same_screen YES

ButtonPress event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291044, (106,47), root:(144,149),
    state 0x10, button 5, same_screen YES

ButtonRelease event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291044, (106,47), root:(144,149),
    state 0x1010, button 5, same_screen YES

comment:8 Changed 15 months ago by Antoine Martin

FWIW: wmi exposes input device information:

import wmi
w = wmi.WMI()
[(x.Name, x.Description, x.HardwareType, x.PointingType) for x in list(w.InstancesOf("Win32_PointingDevice"))]

For more information on device fields, see WMI Tasks: Computer Hardware

comment:9 Changed 14 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:10 Changed 11 months ago by Antoine Martin

xorg ML response from input maintainer Peter Hutterer: emit "smooth scrolling/pixel perfect" events: one option is to emulate a trackstick-like device and send a middle button mouse press before sending scroll events (and relative events otherwise). In libinput this will scroll automatically, in evdev it's a configuration script away (EmulateWheel and EmulateWheelButton). And that scroll type is 'smooth' (at least in libinput, for evdev see https://bugs.freedesktop.org/show_bug.cgi?id=51533)

The other option is to emulate a touchpad and forward the touches as-is. Then you let userspace take care of converting 2 finger to scroll, etc.
That works for android devices with screen, but obviously not for others.

Last edited 7 months ago by Antoine Martin (previous) (diff)

comment:11 Changed 11 months ago by Antoine Martin

Milestone: 1.03.0

Out of time.

comment:12 Changed 7 months ago by Antoine Martin

Milestone: 3.02.1

Changed 7 months ago by Antoine Martin

Attachment: main_v2.c added

uinput example

comment:13 Changed 7 months ago by Antoine Martin

New thread about the same issue: Virtual Touch Events for XTEST.
Includes this code sample: attachment/ticket/173/main_v2.c.

comment:14 Changed 6 months ago by Antoine Martin

This will also help handling javascript wheel events better, see ticket:1424#comment:3

comment:16 Changed 6 months ago by Antoine Martin

Status: acceptedassigned

See events client side: monitoring events (keyboard, mouse) in X using:

xinput test-xi2 --root
Last edited 5 months ago by Antoine Martin (previous) (diff)

Changed 5 months ago by Antoine Martin

Attachment: xi-events.patch added

handle XI2 events

Changed 5 months ago by Antoine Martin

Attachment: xi-events-v2.patch added

updated patch for r15792

comment:17 Changed 5 months ago by Antoine Martin

More links:

Maybe we want to ensure that the keyboard layout is set per device and does not change: New udev property: XKB_FIXED_LAYOUT for keyboards that must not change layouts (but how do we set this without udev?)

Preparatory work in r15790 + r15792. The patch attached allows us to receive XINPUT2 events: raw touch, keyboard and pointer events.
Still TODO:

  • make sure the XI events have all the attributes we need (we'll need to inject the gtk.Window wrappers somehow)
  • augment the packets to include the optional device-id we now have
  • disable regular events when we have XI events enabled
  • enumerate the devices and send them as part of the hello packet, send updates when the device topology changes
  • and the big one: create the matching uinput devices on the server side
  • other plaforms: also enumerate the devices and include the device id
Last edited 5 months ago by Antoine Martin (previous) (diff)

comment:18 Changed 5 months ago by Antoine Martin

More useful pointers:

Changed 5 months ago by Antoine Martin

Attachment: xi-events-v3.patch added

updated patch with device discovery

comment:19 Changed 5 months ago by Antoine Martin

This bug may prevent us from using synthetic events: XSendEvent() fails on XInput2 events

Other problems:

  • device information and events have float values - those won't work with the bencoder
  • raw events are better for us: they don't have acceleration, but the raw events also don't have the window ID in the event structure
  • cooked events have the window ID, but it's the window ID of the window manager, not our window...

To solve both, we could record all XI2 events until we get a "regular" event and then use the extra data. The problems with this approach if that the raw pointer / scroll events are fine grained and may not fire a "regular" event until much later... and some may be discarded altogether.

Last edited 5 months ago by Antoine Martin (previous) (diff)

Changed 5 months ago by Antoine Martin

Attachment: xi-events-v4.patch added

updated patch for r15794

Changed 5 months ago by Antoine Martin

Attachment: xi-events-v5.patch added

updated patch

comment:20 Changed 5 months ago by Antoine Martin

New issues:

  • some other packets send the pointer position update (ie: window configure), those should now also provide the device id... except we don't have it since the event may not be triggered by a device event - PITA but we can probably just use the fallback code for this case without causing problems
  • wayland doesn't send us XI events unless the pointer is in one of our windows (by design) - not a problem, saves us parsing events for nothing and then filtering them
  • wayland sends all mouse events through a single "xwayland-pointer" slave device, making it useless for testing distinct device behaviour
  • the way XI2_Window overrides events needs to be generalized - fugly code
  • the XI2 event parser should provide the modifiers state (we copy it from the regular event for now)
  • we should watch for structure events and only update the list of windows we query when we get an event (and we may also need to stop before the root window with some window managers?)
  • trying to trigger the smooth scrolling using a laptop trackpad over an xterm window, all I see are lots of XI_Motion events with the occasional XI_ButtonPress + XI_ButtonRelease pair when we reach the threshold for a mousewheel event - this could be caused by wayland? (I was expecting touch events... but maybe we need raw events here?)

Changed 5 months ago by Antoine Martin

Attachment: xi-events-v7.patch added

better approach: register event handlers directly with the XI2 helper class

comment:21 Changed 5 months ago by Antoine Martin

With the updated patch, we get the correct events for pointer motion and clicks (keyboard events remain to be done).

Now, the hard part: setting up the server side devices.

If we use uinput, we should ship a file in /etc/X11/xorg.conf.d/ to skip our virtual devices by default, to avoid creating loops when the client is running locally - we can match the device in an inputclass section using MatchVendor and then set Option "Ignore" "true"

The problem with uinput is that as raw as the xinput2 events are, they're not from the same level as uinput.. so some interpretation / un-translation is still going to be needed to re-inject them in such a way that libinput will generate the exact events we want.

But maybe we don't want to use uinput + libinput at all?
We could also write our own input driver and feed it the events directly?
This way we don't need elevated permissions to talk to /dev/uinput and we can just create our own private input socket.
How would that work with dynamic devices though?

Changed 5 months ago by Antoine Martin

Attachment: xi-events-v8.patch added

updated patch: export all device properties

Changed 4 months ago by Antoine Martin

Attachment: xi-events-v9.patch added

updated patch: server know knows details of the device used for some events

comment:22 Changed 4 months ago by Antoine Martin

With the latest patch, the server has all the information it needs about the input devices (device changes need to be propagated - easy) and the input events include the device id (most of them - easy to add).

As for simulating the devices on the server, these examples may help:

The problem is that those examples work on "evdev" devices.. and we get the device information from the X11 server using the "xinput2" API... which is mapped from the kernel's evdev devices via the "libinput" or the "evdev" Xorg driver layer.
It should be possible to retrieve the corresponding evdev device that libinput / evdev uses, but the events we'll receive from the X11 server will be "xinput2" events...

Changed 4 months ago by Antoine Martin

Attachment: 00-ignore-xpra-devices.conf added

tell Xorg to always ignore the virtual input devices we create

Changed 4 months ago by Antoine Martin

Attachment: 00-only-xpra-devices.conf added

tell Xdummy to add our virtual input devices

Changed 4 months ago by Antoine Martin

Attachment: 61-evdev-xpra.rules added

change the udev permissions on input devices

Changed 4 months ago by Antoine Martin

Attachment: xi-events-v10.patch added

use uinput to inject events

comment:23 Changed 4 months ago by Antoine Martin

Installing:


udev will create input devices that a non-root Xorg can access. (we should probably use group access instead of 666 permisions, and restrict the changes to our virtual input devices, not all input devices... see udev rules: controlling permissions and ownership)
The default Xorg server will ignore our virtual input devices:

/usr/libexec/gdm-x-session[9751]: (II) config/udev: Adding input device Fake xpra pointer (/dev/input/event17)
/usr/libexec/gdm-x-session[9751]: (**) Fake xpra pointer: Ignoring device from InputClass "Xpra devices should not be used automatically"

but Xdummy will add and enable them:

[ 96341.568] (II) config/udev: Adding input device Fake xpra pointer (/dev/input/event17)
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "evdev pointer catchall"
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "libinput pointer catchall"
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "All devices"
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "Xpra devices should not be used automatically"
[ 96341.568] (II) Using input driver 'libinput' for 'Fake xpra pointer'
[ 96341.568] (**) Fake xpra pointer: always reports core events
[ 96341.568] (**) Option "Device" "/dev/input/event17"
[ 96341.568] (**) Option "_source" "server/udev"
[ 96341.569] (II) event17 - (II) Fake xpra pointer: (II) is tagged by udev as: Mouse
[ 96341.569] (II) event17 - (II) Fake xpra pointer: (II) device is a pointer
[ 96341.569] (II) event17 - (II) Fake xpra pointer: (II) device removed
[ 96341.584] (**) Option "config_info" "udev:/sys/devices/virtual/input/input68/event17"
[ 96341.584] (II) XINPUT: Adding extended input device "Fake xpra pointer" (type: MOUSE, id 6)
[ 96341.584] (**) Option "AccelerationScheme" "none"
[ 96341.584] (**) Fake xpra pointer: (accel) selected scheme none/0
[ 96341.584] (**) Fake xpra pointer: (accel) acceleration factor: 2.000
[ 96341.584] (**) Fake xpra pointer: (accel) acceleration threshold: 4

With this in place, I can simulate devices via uinput and only Xdummy sees the events.
This can be verified easily with:

xpra info  | grep -i cursor.position

For injecting events, maybe python-evdev will be more useful than python-uinput? (at least the capabilities are better documented)

With the latest patch, I can inject relative motion... but not absolute. (mouse pointers don't do absolute)
Looks like this needs more options to be able to configure the virtual devices properly (see min==max "kernel bug"):

[107288.819] (II) config/udev: Adding input device Logitech M705 - xpra pointer (/dev/input/event17)
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "evdev pointer catchall"
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "libinput pointer catchall"
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "All devices"
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "Xpra devices should not be used automatically"
[107288.819] (II) Using input driver 'libinput' for 'Logitech M705 - xpra pointer'
[107288.819] (**) Logitech M705 - xpra pointer: always reports core events
[107288.819] (**) Option "Device" "/dev/input/event17"
[107288.819] (**) Option "_source" "server/udev"
[107288.820] (II) event17 - (II) Logitech M705 - xpra pointer: (II) is tagged by udev as: Mouse
[107288.820] (EE) event17 - (EE) Logitech M705 - xpra pointer: (EE) kernel bug: device has min == max on ABS_X
[107288.820] (II) event17 - (II) Logitech M705 - xpra pointer: (II) was rejected
[107288.844] (II) event17 - not using input device '/dev/input/event17'.
[107288.844] (EE) libinput: Logitech M705 - xpra pointer: Failed to create a device for /dev/input/event17
[107288.844] (EE) PreInit returned 2 for "Logitech M705 - xpra pointer"
[107288.844] (II) UnloadModule: "libinput"
[107288.845] (II) config/udev: Adding input device DLL0704:01 06CB:76AE Touchpad - xpra pointer (/dev/input/event18)
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "evdev pointer catchall"
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "libinput pointer catchall"
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "All devices"
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "Xpra devices should not be used automatically"
[107288.845] (II) Using input driver 'libinput' for 'DLL0704:01 06CB:76AE Touchpad - xpra pointer'
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: always reports core events
[107288.845] (**) Option "Device" "/dev/input/event18"
[107288.845] (**) Option "_source" "server/udev"
[107288.847] (II) event18 - (II) DLL0704:01 06CB:76AE Touchpad - xpra pointer: (II) is tagged by udev as: Mouse
[107288.847] (EE) event18 - (EE) DLL0704:01 06CB:76AE Touchpad - xpra pointer: (EE) kernel bug: device has min == max on ABS_X
[107288.847] (II) event18 - (II) DLL0704:01 06CB:76AE Touchpad - xpra pointer: (II) was rejected
[107288.872] (II) event18 - not using input device '/dev/input/event18'.
[107288.872] (EE) libinput: DLL0704:01 06CB:76AE Touchpad - xpra pointer: Failed to create a device for /dev/input/event18
[107288.872] (EE) PreInit returned 2 for "DLL0704:01 06CB:76AE Touchpad - xpra pointer"
[107288.872] (II) UnloadModule: "libinput"

Changed 4 months ago by Antoine Martin

Attachment: xi-events-v11.patch added

switch to using python-evdev

comment:24 Changed 4 months ago by Antoine Martin

Milestone: 2.12.2

Mostly merged in r15802 (big), this doesn't do anything useful yet and is not enabled unless the client specifies --input-devices=xi.

With "xi" enabled, the server gets a list of devices, which is not used for anything yet, and the pointer packets include the device responsible for the event (as well as more details on the event), ie:

process_mouse_common from device=Logitech G400s Optical Gaming Mouse

This will have to do for this release, at least the xinput code is merged and won't bitrot too much.
Enabling uinput devices for touch events only should work, but the better approach is to write a new xorg input driver so we can inject the events exactly as we receive them.

comment:25 Changed 4 months ago by Antoine Martin

This may help with MS Windows to get the device ID and layout for each keyboard device:

$ python -c "from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_Keyboard')
print([(x.Properties_['DeviceID'].Value, x.Properties_['Layout'].Value) for x in res])
[(u'ACPI\\PNP0303\\4&1D401FB5&0', u'00000809')]
Last edited 4 months ago by Antoine Martin (previous) (diff)

Changed 7 weeks ago by Antoine Martin

Attachment: uinput-server.patch added

implement pointer as a uinput device

comment:26 Changed 7 weeks ago by Antoine Martin

Milestone: 2.23.0

Moved the uinput server-side support to #1611.

We can keep this ticket as a tracker.

comment:27 Changed 3 weeks ago by Antoine Martin

r16721 uses xinput2 devices on posix clients by default.

Note: See TracTickets for help on using tickets.