Xpra: Ticket #770: drag and drop support

At the very least, we should support drag and drop within the same application, since all this stays server side.

If we do support some form of remote to local drag and drop (ie: dropping a URL on a browser), we may want to include some form of filtering, like the clipboard filter file.

I'm not sure if we want to use GTK for this, it may just end up getting in the way, just like it did with clipboard - with which it shares a lot of concepts (selections, conversions, etc). Still useful for a quick test application.

Some pointers:



Mon, 05 Jan 2015 07:16:50 GMT - Antoine Martin: owner, status, description changed


Mon, 05 Jan 2015 07:30:32 GMT - Antoine Martin: attachment set

a simple Java app which accepts drops and shows the flavors available


Wed, 14 Jan 2015 10:05:16 GMT - Antoine Martin: attachment set

updated example code


Wed, 14 Jan 2015 12:41:35 GMT - Antoine Martin:

Right, so this works... but only some of the time!

x11trace (excerpts):

Looks like the DND code picks the wrong window sometimes (ie below: the xterm I launch from), and then decides that DND is not supported. This probably comes from QueryTree.

Which looks like it is coming from or somewhere near http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/awt/X11/XlibUtil.java's getChildWindows (which uses a set, which is not ordered and would explain the randomness aspect).


Thu, 15 Jan 2015 08:22:52 GMT - Antoine Martin:

After building Java from source with home made patches to add extra logging, I can see:

I think TILS are:

INFO: updateTargetWindow(XMotionEvent = type = MotionNotify, serial = 658, send_event = false, display = 140270009602512, window = 44, root = 68, subwindow = 4194347, time = 8290765, x = 95, y = 105, x_root = 95, y_root = 105, state = 272, is_hint = 0, same_screen = true, ) dragProtocol=null, targetRootSubwindow=0, subwindow=40002b
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(40002b) children=[40002c, 40003b, 400447]
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(40002c) children=[]
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(..) not found!
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(40003b) children=[600022]
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(600022) is true toplevel
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(..)=600022
Jan 15, 2015 1:07:31 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(..)=600022

vs

INFO: updateTargetWindow(XMotionEvent = type = MotionNotify, serial = 630, send_event = false, display = 140602735480256, window = 44, root = 68, subwindow = 4194347, time = 8015604, x = 83, y = 120, x_root = 83, y_root = 120, state = 272, is_hint = 0, same_screen = true, ) dragProtocol=null, targetRootSubwindow=0, subwindow=40002b
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(40002b) children=[40002c, 4003d6, 40003b]
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(40002c) children=[]
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(..) not found!
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(4003d6) children=[800007]
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(800007) is true toplevel
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(..)=800007
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer findClientWindow
INFO: findClientWindow(..)=800007
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer doUpdateTargetWindow
INFO: doUpdateTargetWindow(40002b, 8015604) attached using sun.awt.X11.XDnDDragSourceProtocol@23b2a3b8
Jan 15, 2015 1:02:55 PM sun.awt.X11.XDragSourceContextPeer updateTargetWindow
INFO: sendEnterMessage

I am stumped. (and also found a bug with xkbmap_mod_pointermissing... see r8479)


Thu, 15 Jan 2015 12:27:57 GMT - Antoine Martin:

Since Java calls findClientWindow on the window it gets from the XMotionEvent, I thought that maybe we were somehow using the wrong window when generating the mouse motion events. So the patch above replaces the GTK calls with relative motion events, which could be useful for #137 and may be worth applying in any case, after cleaning it up. But that doesn't seem to change the window id that Java sees, and it then still picks the wrong window half the time!

This patch is really ugly and broken, we can probably skip a lot of the messy logic and just rely on X11Keyboard.window_at_pointer to return None if we're not in the window we are looking for? Needs testing with OR windows, etc.

It is worth mentioning that GTK's window_at_pointer does not seem to work properly, it doesn't seem to be aware of the window size changes... Which could be causing us other problems elsewhere (focus, geometry, events..).


Thu, 15 Jan 2015 12:31:57 GMT - Antoine Martin: attachment set

hackish patch for testing relative motion events


Fri, 16 Jan 2015 10:14:44 GMT - Antoine Martin: owner, status changed

Well, in the end, I refused to let this bug go, I just could not see any way of getting Java to not traverse the window tree all the way up to the world window (which then causes it to pick the wrong window as target), so r8484 completely changes the way windows are organized: they are now all children of the root window and the "world window" is just another window. This matches what other window managers do, and does not seem to have any other negative impacts. It is also cleaner / makes more sense.

I guess it could be backported to v0.14.x eventually, after a long time settling in the 0.15 branch.

afarr: this one is a BIG FYI, it is a very significant change. Which could potentially cause us some focus problems (or maybe even fixes for existing problems!).


Fri, 16 Jan 2015 10:18:16 GMT - Antoine Martin: attachment set

extra logging that I've used to help diagnose what Java is doing


Fri, 23 Jan 2015 01:20:58 GMT - alas: status changed; resolution set

Tested (perhaps redundantly) successfully with windows 0.15.0 r8522 and OSX 0.15.0 r8527 against fedora 20 0.15.0 r8527 - quite a success (I remember this was breaking browsers at one point).

Nice work!

Closing (but I'll watch for focus issues that might relate).


Mon, 09 Feb 2015 07:30:13 GMT - Antoine Martin:

This change has now been in trunk for a month and nothing seems to have been broken by it, so I have backported it in r8644.


Sat, 14 Feb 2015 04:34:51 GMT - Antoine Martin: priority, status, milestone changed; resolution deleted

Just got an email from Calvin saying that this was causing problems in 0.14.x, so it looks like this will need to be reverted before the release.


Mon, 16 Feb 2015 18:02:58 GMT - alas:

It's odd...

I'll see about pinning this down some more (I really would rather not disable drag and drop if possible).


Thu, 19 Feb 2015 22:23:53 GMT - alas:

OK, confirmed... I'm seeing drop menu issues with trunk builds on fedora 20 and 21 servers, with both windows and osx clients. The issues seem to be different from browser to browser (what a surprise!), but the same between fedora 20 servers and fedora 21 and between osx and windows clients.

windows client (8.1) 0.15.0 r8647 (smo's build) osx client 0.15.0 r8640 (my build) fedora 20 0.15.0 r8661 (your beta build) fedora 21 0.15.0 r8661 (also your beta build)

I grabbed some logs, both with windows and osx clients... will attach probably (hopefully) more than you need.

The interesting part, as far as I can tell, from the windows connection to the fedora 21 server using epiphany is this:

2015-02-19 12:43:40,411 _button_action(1, <gtk.gdk.Event at 076BF530: GDK_BUTTON_PRESS x=801.00, y=354.00, button=1>, True) wid=5 / focus=5, pointer=(2081, 354), modifiers=['mod2'], buttons=[]
2015-02-19 12:43:40,443 process_new_common: [17, 1946, 348, 199, 173, {'opacity': -1, 'fullscreen': False, 'has-alpha': True, 'xid': '0xa00cb3', 'pid': 19251, 'window-type': ('POPUP_MENU',), 'maximized': False, 'override-redirect': True}], OR=True
2015-02-19 12:43:40,443 make_new_window(..) client_window_classes=[<class 'xpra.client.gl.gtk2.gl_client_window.GLClientWindow'>, <class 'xpra.client.gtk2.border_client_window.BorderClientWindow'>], group_leader_window=<gtk.gdk.Window object at 0x77d6e90 (GdkWindow at 0x22f29a0)>
2015-02-19 12:43:40,443 GLClientWindow(..)
2015-02-19 12:43:40,443 <class 'xpra.client.gl.gtk2.gl_client_window.GLClientWindow'>(gtk2.client, <gtk.gdk.Window object at 0x77d6e90 (GdkWindow at 0x22f29a0)>, 17, 1946, 348, 199, 173, {'opacity': -1, 'fullscreen': False, 'has-alpha': True, 'xid': '0xa00cb3', 'pid': 19251, 'window-type': ('POPUP_MENU',), 'maximized': False, 'override-redirect': True}, True, {}, (0, 0))
2015-02-19 12:43:40,443 set_window_type(['POPUP_MENU']) hints=9
2015-02-19 12:43:40,443 new_backing(199, 173) backing_class=<class 'xpra.client.gl.gtk2.gl_window_backing.GLPixmapBacking'>
2015-02-19 12:43:40,443 make_new_backing(<class 'xpra.client.gl.gtk2.gl_window_backing.GLPixmapBacking'>, 199, 173) effective backing class=<class 'xpra.client.gl.gtk2.gl_window_backing.GLPixmapBacking'>, server alpha=True, window alpha=False
2015-02-19 12:43:40,443 Win32Hooks: window frame size is 8x8
2015-02-19 12:43:40,443 failed to set group leader: 'module' object has no attribute 'SHGetPropertyStoreForWindow'
2015-02-19 12:43:40,443 GL do_configure_event(<gtk.gdk.Event at 0758A050: GDK_CONFIGURE x=1946, y=348, width=199, height=173>)
2015-02-19 12:43:40,443 GL do_configure_event(<gtk.gdk.Event at 0758A050: GDK_CONFIGURE x=1946, y=348, width=199, height=173>)
2015-02-19 12:43:40,443 GLClientWindow(17 : gtk2.GLWindowBacking(17, (199, 173), None)).do_map_event(<gtk.gdk.Event at 0758A050: GDK_MAP>) OR=True
2015-02-19 12:43:40,443 GL do_configure_event(<gtk.gdk.Event at 0758A050: GDK_CONFIGURE x=1946, y=348, width=199, height=173>)
2015-02-19 12:43:40,520 do_motion_notify_event(<gtk.gdk.Event at 0758A050: GDK_MOTION_NOTIFY x=135.00, y=7.00>) wid=17 / focus=5, pointer=(2081, 355), modifiers=['mod2'], buttons=[1]
2015-02-19 12:43:40,614 _button_action(1, <gtk.gdk.Event at 076BFF50: GDK_BUTTON_RELEASE x=135.00, y=7.00, button=1>, False) wid=17 / focus=5, pointer=(2081, 355), modifiers=['mod2'], buttons=[1]
2015-02-19 12:43:40,614 button action: simulating a missing mouse-down event for window 5 before sending the mouse-up event
2015-02-19 12:43:40,927 do_motion_notify_event(<gtk.gdk.Event at 076BFE30: GDK_MOTION_NOTIFY x=134.00, y=8.00>) wid=17 / focus=5, pointer=(2080, 356), modifiers=['mod2'], buttons=[]
2015-02-19 12:43:40,943 do_motion_notify_event(<gtk.gdk.Event at 076BFE30: GDK_MOTION_NOTIFY x=133.00, y=9.00>) wid=17 / focus=5, pointer=(2079, 357), modifiers=['mod2'], buttons=[]

Thu, 19 Feb 2015 22:31:39 GMT - alas: attachment set

click on drop menu in epiphany, drop menu non-selectable


Thu, 19 Feb 2015 22:32:21 GMT - alas: attachment set

click on drop menu in firefox, try to select with a second click, non-responsive


Thu, 19 Feb 2015 22:33:02 GMT - alas: attachment set

click and hold on drop menu in firefox, select by un-click, drop menu works as expected


Thu, 19 Feb 2015 22:34:57 GMT - alas: attachment set

osx client click-hold epiphany, drop menu non-interactive


Thu, 19 Feb 2015 22:36:36 GMT - alas: attachment set

osx client click-hold firefox, drop menu behaves as expected


Thu, 19 Feb 2015 22:39:02 GMT - alas:


Fri, 20 Feb 2015 02:29:52 GMT - alas:

Testing again with windows client 0.14.20 r8634 (your beta) against fedora 20 0.14.20 r8645 (also your beta) ... I find the behavior (both firefox and epiphany) to be the same as the 0.15.0 trunk builds tested above.


Sat, 21 Feb 2015 15:17:59 GMT - Antoine Martin:

(..) Apparently we had some trunk code in our 14.19 server/clients?) Testing again with windows client 0.14.20 r8634 (your beta) against fedora 20 0.14.20 r8645 (also your beta) ... I find the behavior (both firefox and epiphany) to be the same as the 0.15.0 trunk builds tested above.


This change has been backported to the v0.14.x branch (but not released yet), see comment:7. I hope we can find a way of fixing this problem without reverting it...

Can you please clarify the difference between the two firefox: click + release + move vs click + hold + move?

I'm trying to find a simple example to test against with epiphany. But for example: w3schools' tryhtml_option_selected works fine here. I've also tried the steps from comment:11 (apple's site), still no luck!


Mon, 23 Feb 2015 21:47:41 GMT - alas:

Uhhmmm... testing again (to find a simple example with epiphany)... I can't repro the problem anymore - neither with epiphany nor with firefox... neither on the locate.apple.com nor on facebook.com sign in drop downs. Very odd.

Win32 client 0.15.0 r8689 (windows 8.1) Fedora 20 server 0.15.0 r8661 (same as before).

Just in case, though, I'll try to clarify my made-up nomenclature. When clicking on a drop down, one can either simply click, or one can click and hold the button depressed.

For a time I was finding that, with firefox, if I just left clicked a drop down then moved the cursor over a selection and tried to left click to select - it was failing to select (click + release + move).

Meanwhile, if I left clicked but held the button depressed, then moved the cursor to a selection and un-clicked to select - it succeeded at selecting (click + hold + move).

With the new windows client though, either method is working as expected with either epiphany or firefox. (Could the fixes for the unicode errors mentioned in #786, or the SHGetPropertyStoreForWindow issues suspected related to #799 have "accidentally" fixed this issue?)


Tue, 24 Feb 2015 00:55:06 GMT - Antoine Martin:

Just in case, though, I'll try to clarify my made-up nomenclature. When clicking on a drop down, one can either simply click, or one can click and hold the button depressed.


OK, I thought that was how it worked (the description was adequate) but since I was having problems reproducing it, I was trying to make sure I wasn't just doing it wrong!


With the new windows client though, either method is working as expected with either epiphany or firefox. (Could the fixes for the unicode errors mentioned in #786, or the SHGetPropertyStoreForWindow issues suspected related to #799 have "accidentally" fixed this issue?)


I doubt it, but since this change is potentially disruptive, I am going to keep this ticket open and delay 0.14.20 until we've had a chance to test this some more. How did Calvin originally reproduce the problem to be able to narrow it down to this changeset? (maybe re-assign to him?)


Fri, 27 Feb 2015 16:12:51 GMT - Antoine Martin:

Any move on this? Can you ask Calvin how he identified this changeset as problematic?


Tue, 03 Mar 2015 02:20:18 GMT - Antoine Martin:

r8737 (+backport in r8738) allows us to toggle the implementation using the env var:

XPRA_REPARENT_ROOT=0 xpra start ...

I think this is safer as it gives us a way to go back to the previous behaviour if we have to.


Wed, 04 Mar 2015 01:08:58 GMT - alas:

Firstly, to answer the how/why of Calvin's identification and change... it was the latest changeset and largely a guess that, since the drop menus had been working before, they would (maybe?) work again if that change were reverted.

As to what I'm seeing investigating some more... it looks like the bugs I had been seeing in the drop down menu windows is now confined to OSX clients. On the other hand, while the XPRA_REPARENT_ROOT=0 environment variable seems to work for OSX, it horrendously breaks the win32 clients (at least, it does so with firefox & epiphany).

(locate.apple.com - clicking and holding the drop down to select an item from either the sales or the service button item select drop menus, followed by releasing the click to select works as expected - but clicking & releasing, then trying to click again on a selection from the drop down menu displayed not only fails to select anything from the drop down menu, but seems to also be interpreted as a "held click", highlighting anything subsequently moused over.)

(Similarly, starting a gedit application and trying to use the application menu to select Search, then trying to select Replace from the displayed drop down menu fails ... though it fails the same whether the click is held and un-clicked to select or if it is released and a second click is attempted to make a selection from the menu.)


Thu, 05 Mar 2015 04:47:35 GMT - Antoine Martin:

.. while the XPRA_REPARENT_ROOT=0 environment variable seems to work for OSX, it horrendously breaks the win32 clients (at least, it does so with firefox & epiphany).


I don't understand this. Setting this environment variable only effectively reverts the changeset and goes back to the pre r8484 behaviour. Please clarify and confirm that setting the env var to 0 gives the same behaviour as in 0.14.x. If it does not, then we are dealing with something else - possibly related.


Fri, 06 Mar 2015 01:50:08 GMT - alas: owner, status changed

Well, nevermind - it looks like the issue with our win32 build (last issue with #799) tripped me up for this testing... with the fixed window client the XPRA_REPARENT_ROOT=0 environment variable is working as expected.

Passing it back.


Sun, 08 Mar 2015 17:04:31 GMT - Antoine Martin: status changed

Until I can investigate with OSX (virtualbox is having problems), I have changed the default value of XPRA_REPARENT_ROOT in r8763 for trunk and r8764 for v0.14.x

So we have to use "1" to enable reparent to root. (the default behaviour is now as it was before this change)


Mon, 09 Mar 2015 11:05:25 GMT - Antoine Martin:

As far as I can tell, the http://locate.apple.com/ widgets are just regular select HTML elements.

This is a much better and simpler test case for the drop down (with source visible): w3schools: try select form


when I try to click on the two right-most (Service and Consulting), the clicks register on the left most tiles (Sales and Training) also... enabling a click on one of the "right-most" tiles to put keyboard focus in a textbox on a left-most tile..


If I understand this correctly, this looks like a completely different issue. Sounds like the dummy driver is not patched. Please confirm that this is related to the reparent code, and if not create a new ticket for this issue.


Mon, 09 Mar 2015 11:09:24 GMT - Antoine Martin:

In fact, we have a minimal test program already named "test drop down", just for this purpose - 37 lines all included . A million times better than testing with a browser.

Could well be related to #469. See also #713.


Sun, 18 Oct 2015 13:11:25 GMT - Antoine Martin: owner, status changed

If this is still an issue, please provide simple reproduction steps (ie: preferably not using a browser)


Sat, 24 Oct 2015 01:13:12 GMT - alas: status changed; resolution set

Well, I can't seem to reproduce this with a browser anymore - so rather than trying to reproduce on something more simple, I'm going to conclude that it is no longer a problem.

Closing.


Sat, 24 Oct 2015 03:03:07 GMT - Antoine Martin: status changed; resolution deleted

Re-opening as I need to make absolutely certain.

Did you test both with a server started with: XPRA_REPARENT_ROOT=1 ? The default got changed back in comment:21. Also please make sure you test with an osx client post r10983 as this bug would prevent some metadata from applying, potentially interfering with the testing.


Mon, 26 Oct 2015 23:29:26 GMT - alas: status changed; resolution set

Thanks for that heads up.

Tested again with XPRA_REPARENT_ROOT=1

0.16.0 r10972 fedora 21 server, 0.16.0 r11015 osx client, 0.16.0 r10983 windows client (on 8.1).

Can't find any sign of the issues in this ticket - closing again.


Tue, 27 Oct 2015 02:55:08 GMT - Antoine Martin:

r11032 makes XPRA_REPARENT_ROOT=1 the default.

FYI: there is already a ticket for firefox clipboard weirdness which may be relevant: #883


Thu, 19 Nov 2015 13:23:26 GMT - Antoine Martin: status changed; resolution deleted

This causes regressions like ticket:911#comment:3, so this is reverted in r11292.

We'll also need to make sure we don't cause regressions with #814.


Thu, 19 Nov 2015 13:24:03 GMT - Antoine Martin: owner, status, milestone changed

too late to deal with this, re-scheduling


Mon, 01 Feb 2016 18:02:31 GMT - urzds: cc set


Wed, 16 Mar 2016 05:40:08 GMT - Antoine Martin: status, milestone changed


Tue, 12 Jul 2016 16:52:22 GMT - Antoine Martin: milestone changed

Milestone renamed


Tue, 13 Sep 2016 15:33:59 GMT - Antoine Martin: owner, status changed

Following some focus fixes in r13600, r13690 re-enables "reparent to root" again... hopefully we'll have better luck this time. Seems to work so far.

Note: this may cause regressions when connecting a new client to an old server and vice versa. (we can re-add some conditional workarounds via capabilities later on - just make a note of them for now but this is not the priority for testing, new to new is)

Tickets to (re)check: #814, #469, #911


Sat, 17 Sep 2016 01:48:00 GMT - alas: owner changed

Ok... testing with 1.0 r13763 osx client (10.9) against 1.0 r13767 fedora 23 vm.

Tried all the old-timey permutations I could think of with the locate.apple.com and the w3-schools link you provided, with both chrome and especially firefox (epiphany?, meh... don't care so much anymore)... both with a mouse and a tracpad. All looks well & good.

Tested the above with the default, which should be enabling "reparent to root" (and in the case of the issues mentioned in this ticket, also with XPRA_REPARENT_ROOT=1, which disables it again?... or maybe it should be tested with =0 and compared?).

In any case, it looks like you've successfully implemented "reparent to root" without breaking anything newly, though it would've been nice if it had fixed the "click-through" behavior of #469 as well.

Handing this back to you... I think I tested all the things that you were looking for and you can close this (though not #469), but let me know if I overlooked something.


Sat, 17 Sep 2016 05:38:39 GMT - Antoine Martin: status changed; resolution set

Thanks!

I believe r13600 caused at least one bad regression reported in #1307 (server crash), now fixed in r13769. (you needed to disconnect lots of times to hit it - and maybe running something like gtkperf at the same time helps to hit that codepath)

Closing as we can follow up in #469 and I don't think we will need version specific toggles. (will just need to remember to re-test with older clients / servers before the final release)


Thu, 02 Mar 2017 12:01:01 GMT - Antoine Martin: status changed; resolution deleted

This feature has caused another regression: gedit running on centos 6 with osx client... (see ticket:1453#comment:1), and so the default has been reverted again in the v1.0.x branch.

I am hoping that we can fix this client-side and re-enable drag-n-drop in v1.0.x. It's likely to be caused by focus / window id being reported wrong by the macosx client.


Fri, 10 Mar 2017 12:04:35 GMT - Antoine Martin: status changed; resolution set

See ticket:1453#comment:2, will follow up in #469 (re-opened).


Mon, 10 Apr 2017 07:27:55 GMT - Antoine Martin:

Follow up in #1493 for proper drag-n-drop support.


Tue, 30 Oct 2018 06:45:18 GMT - Antoine Martin:

And another regression: #1999


Sat, 23 Jan 2021 05:05:27 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/770