Proposal on enhancing MouseWheelEvent to include absolute scrolling delta
Hi All,
Next follows a proposal on enhancing MouseWheelEvent to include absolute
scrolling delta.
My name is Pavel Fatin, I work at JetBrains (IntelliJ IDEA). Although
this is my first message here, I have been working with Java / AWT /
Swing since Java 1.2 and have some in-depth experience (for example, my
research on low-latency painting in AWT and Swing
<https://pavelfatin.com/low-latency-painting-in-awt-and-swing/>).
Because we at JetBrains receive many requests
<https://youtrack.jetbrains.com/issue/IDEA-76396> to implement
smooth-scrolling in IDEA, as an experiment, we extended
<https://github.com/JetBrains/intellij-community/commit/34b9dfd0585937c3731e06a89554d1dc86f7f235>
our custom scroll pane class to handle MouseWheelEvent's
getPreciseWheelRotation()
<https://docs.oracle.com/javase/8/docs/api/java/awt/event/MouseWheelEvent.html#getPreciseWheelRotation-->
data (that method was introduced in Java 7, but it’s not yet used by AWT
/ Swing directly). As a result, the scrolling "smoothness" and precision
improved substantially, however on OS X it was still a far cry from the
native scrolling experience, particularly:
*
Scrolling speed was unit-dependent (while in OS X it's not). The
concept of "wheel rotation" is inherently tied to the idea of
"scrolling unit" (usually 1 or 3 lines). This makes sense for a real
mouse scrolling wheel, but doesn't make much sense for a touchpad –
one expects a uniform mapping between physical touch gesture and a
screen update, consistent across different applications. The same
goes for touchscreen interfaces. Introduction of the fractional
wheel rotations increases precision but still retains the
"rotation-unit" bound.
*
Spatial precision was lacking. Typical scrolling deltas were in
range 0.5..>10, and this was only a slight improvement over 1..>10
deltas.
*
Temporal resolution was lacking, so the scrolling was rather
coarse-grained and there was a substantial lag between starting a
gesture and a reaction on the screen. This mostly stems from the
insufficient spatial resolution. Added interpolation helped to
artificially increase spatial- and temporal resolutions, but
couldn't solve the input lag and the lack of positioning precision.
Wondering how the native OS X applications can perform much better we
started to investigate the Cocoa API. It turns out, that since Mac OS X
10.7 (Lion) NSEvent
<https://developer.apple.com/reference/appkit/nsevent>, in addition to
the deltaX
<https://developer.apple.com/reference/appkit/nsevent/1534871-deltax>
and deltaY
<https://developer.apple.com/reference/appkit/nsevent/1534158-deltay>
introduced scrollingDeltaX
<https://developer.apple.com/reference/appkit/nsevent/1524505-scrollingdeltax>
and scrollingDeltaY
<https://developer.apple.com/reference/appkit/nsevent/1535387-scrollingdeltay>
properties that supply high-resolution scrolling information (with
hasPrecisionScrollingDeltas
<https://developer.apple.com/reference/appkit/nsevent/1525758-hasprecisescrollingdeltas>
property that specifies how to interpret the data). So far so good, but
those properties are quite a different beast – they contain /absolute/
scrolling deltas rather than relative deltas, supplied by the previous
values.
How can we integrate such values in the existing Java API?
One way is to try to somehow translate the absolute deltas to relative
"wheel rotations", but the wheel event per se represents input data and
doesn't deal with possible scrolling units as such. Another problem with
this approach is that OS might provide so-called "pixel-perfect"
scrolling, where smallest delta is guaranteed to be 1 pixel (yet still
preserving large-distance scrolling via OS-level acceleration), as OS
knows real physical resolution of particular input / output devices and
the acceleration curve, and such a translation cannot reliably handle
this. And one more problem with the masking is compatibility – because
the new data have much more temporal resolution, existing applications
that process precise wheel events synchronously might be overwhelmed,
hog CPU and lag unnecessarily. All in all, "partial wheel rotation" is
an imperfect and "leaky" abstraction for the absolute scrolling delta.
A possible "hack" is to put the absolute values directly into the
relative delta property and then to hardcode unit size to be 1 on Mac OS
(that is what GTK+ 3 does
<https://github.com/GNOME/gtk/blob/master/gtk/gtkscrolledwindow.c#L1245>),
but that is what it is – a hack, not compatible with the existing codebase.
It seems, that the most reasonable solution is to introduce a new,
"absolute" kind of scrolling deltas, in addition to the relative one.
That is, for example, what QWheelEvent <https://goo.gl/kZqxVc> does with
its angleDelta <https://goo.gl/zwxBt9> and pixelDelta
<https://goo.gl/F5A37V> properties. We tried that approach in our
JetBrains Runtime by extending
<https://github.com/JetBrains/jdk8u_jdk/commit/568f2dae82b0fe27b79ce6943071d89463758610>
MouseWheelEvent with getScrollingDelta() method and implementing
<https://github.com/JetBrains/jdk8u_jdk/commit/a3cb8807b148879e9c70a74a8a16c30a28991581>
it in Mac OS – the result is a genuine, OSX-like scrolling experience in
Java.
We chose the method name “getScrollingDelta” (a la Apple) instead of
“getPixelDelta” (a la Qt) because the input event by itself doesn't
necessarily results in display scrolling, and because the concept of
"pixel" is not that precise. Likewise, we chose double type for further
extensibility, considering the case with getWheelDelta() (NSEvent also
uses a floating point value).
It should be noted, that the absolute delta /complements/ rather than
replaces the relative value. Those two values are not interchangeable
and OS generates the values separately (for legacy devices only the
relative delta is generated and, in such cases, we can use 0 as an
absolute one). Addition of the absolute scrolling delta fully preserves
the functioning of Java's existing event processing mechanisms.
Although currently only Mac OS supplies that kind of data, Windows
Precision Touchpad and XInput2 / libinput might follow, as smooth
scrolling becomes a thing nowadays. Additionally, touchscreen-based
scrolling needs precisely this kind of scrolling deltas.
While we may implement this functionality privately in our JetBrains
Runtime, we believe that more people can benefit from it. Moreover, the
enhancement of MouseWheelEvent to support pixel-perfect scrolling can
pave a road for smooth scrolling in AWT and Swing (which is a good thing
to improve usability, both GTK+ 3 and Qt 5 already support that).
--
Sincerely, Pavel
