Handling of simultaneous key-down and mouse-button events
I'm using an application on a remote desktop served by Guacamole 0.9.14 (Remote side: CentOS 7.5, MATE Desktop, TurboVNC 2.2, VirtualGL 2.6). The application uses the following workflow: If pressing the Spacebar (and holding it down) it shows an overlay on an OpenGL window it renders. This overlay has three buttons. Each button selects a different action for the 3D view (rotate, zoom, pan). So while the spacebar is pressed, I'm moving the mouse to this overlay, press the left mouse button to select the action, and then move the mouse (spacebar and left mouse button still pressed at the same time) to perform the action on the 3D view. While a normal TurboVNC+VirtualGL connection works for this workflow, the behavior gets weird when guacamole is on top of it. The overlay flickers once I start moving the mouse (mouse button and spacebar pressed) and the 3D view doesn't perform the expected action. I suppose that guacamole has some difficulty passing the events to the remote session. It seems like it sends a series of key-down and key-release events instead of a single key-down event at the start and a key-release event at the end (at least the behavior of the remote application looks like this). Has anyone seen such a behavior?
Re: Handling of simultaneous key-down and mouse-button events
On Mon, Oct 22, 2018, 08:02 Felix Wolfheimer wrote: > I'm using an application on a remote desktop served by Guacamole 0.9.14 > (Remote side: CentOS 7.5, MATE Desktop, TurboVNC 2.2, VirtualGL 2.6). The > application uses the following workflow: > If pressing the Spacebar (and holding it down) it shows an overlay on an > OpenGL window it renders. This overlay has three buttons. Each button > selects a different action for the 3D view (rotate, zoom, pan). So while > the spacebar is pressed, I'm moving the mouse to this overlay, press the > left mouse button to select the action, and then move the mouse (spacebar > and left mouse button still pressed at the same time) to perform the action > on the 3D view. > > While a normal TurboVNC+VirtualGL connection works for this workflow, the > behavior gets weird when guacamole is on top of it. The overlay flickers > once I start moving the mouse (mouse button and spacebar pressed) and the > 3D view doesn't perform the expected action. I suppose that guacamole has > some difficulty passing the events to the remote session. It seems like it > sends a series of key-down and key-release events instead of a single > key-down event at the start and a key-release event at the end (at least > the behavior of the remote application looks like this). Has anyone seen > such a behavior? > Yes. Unfortunately, this isn't a bug per se, but a platform- and browser-specific aspect of how key events are fired. The key events of printable keys may autorepeat while held. To unify things across platforms, as not all do this (or even reliably report keyup), Guacamole also has its own autorepeat timer within the keyboard handling. Automatically debouncing with client-side heuristics would be another approach which could conceivably be implemented (and that would be compatible with your intentions), but exactly how to do this without breaking the expectations of protocols and applications which rely on automatic key repeat would be tricky. - Mike
Re: Handling of simultaneous key-down and mouse-button events
Hi Mike, thanks for confirming that this is a kind of known behavior. Is there a way to tweak the auto-repeat timer in Guacamole? I suppose that setting it to a high value might give the behavior I'd need for the application although this might lead to unexpected behavior (?) for other applications. signature.asc Description: This is a digitally signed message part
Re: Handling of simultaneous key-down and mouse-button events
On Mon, Oct 22, 2018 at 1:47 PM Felix Wolfheimer wrote: > Hi Mike, > > thanks for confirming that this is a kind of known behavior. Is there a > way to tweak the auto-repeat timer in Guacamole? I suppose that setting > it to a high value might give the behavior I'd need for the application > although this might lead to unexpected behavior (?) for other > applications. Yes. You'd be better off looking into developing some sort of autorepeat debouncing heuristic within Guacamole.Keyboard and contributing that, as it is something which would make sense to have even though the general case should probably assume client-side autorepeat. The main issue is making things work identically across the board (all browsers/platforms) and avoiding breaking the expectations of applications and remote desktop servers. I wouldn't recommend just setting the timer interval to a high value. Besides not working across all platforms, it's a nasty hack. ;) - Mike
Re: Handling of simultaneous key-down and mouse-button events
Hi Mike thanks a lot for your input. I thought a bit about it and it seems that the heuristics which makes sense here is: Use the keyboard behavior as is if no mouse button is pressed. If a mouse button is pressed, don't send repeated key events while the mouse button is pressed. Many CAD programs use combined key-down + mouse button-down + mouse move events to navigate in their 3D views, so I think that this behavior makes sense for at least this class of applications. I looked at the Guacamole.Keyboard as well as Guacamole.Mouse implementations. As I'm not a Javascript programmer I'm not completely sure whether I understand all of this stuff correctly, but it seems to me that I might get the described behavior by changing a function in guacamole/src/main/webapp/app/client/directives/guacClient.js: // Translate local keydown events to remote keydown events if keyboard is enabled $scope.$on('guacKeydown', function keydownListener(event, keysym, keyboard) { if ($scope.client.clientProperties.keyboardEnabled && !event.defaultPrevented && !($scope.client.mouse.currentState.left || $scope.client.mouse.currentState.middle || $scope.client.mouse.currentState.right)) { client.sendKeyEvent(1, keysym); event.preventDefault(); } }); The additional "!($scope.client.mouse.currentState.left || $scope.client.mouse.currentState.middle || $scope.client.mouse.currentState.right)" should tell the client to not send repeated key-down events if any mouse button is clicked. Does this sound right? signature.asc Description: This is a digitally signed message part
Re: Handling of simultaneous key-down and mouse-button events
Hi Mike, found a perfect way to get what I want from Guacamole and just wanted to share what I did in case anyone runs into a similar issue. In guacamole-common-js/src/main/webapp/modules/Keyboard.js there's a list (no_repeat) of keys for which the key-down event is not repeatedly send to the server if a user presses the key and and holds it down. I just added the spacebar key to this list and rebuild the webapp. While the application on the remote side is now working as expected, the drawback is that something like a text box of a remote application also won't get multiple key-down events if the user holds down the space-bar, i.e., if someone wants to type multiple spaces in such a window he/she needs to press the spacebar for each space he/she wnats to type on the remote side. For me that's perfectly fine. :-) Thanks for your help, Mike! signature.asc Description: This is a digitally signed message part