Re: [fullscreen] Problems with mouse-edge scrolling and games
Thank you all. This is wonderfully helpful. On Mon, Feb 24, 2014 at 10:07 AM, Vincent Scheib sch...@google.com wrote: One way to move forward on those topics is to prototype functional solutions on Windows, Mac, and Linux and illuminate a practical solution that browsers may adopt. I did an investigation in 2011 and did not find a solution. For clipping only I believe Mac was the only platform I didn't find a solution for. Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we can implement the functionality, we can discuss how to express this in an API. On Sun, Feb 23, 2014 at 7:18 AM, Brandon Jones bajo...@google.com wrote: - The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. An OS-level visible cursor with pointer lock would be the holy grail. Maybe we can spend some time revisiting the implementations of these to prove out the functionality and push the spec forward a little.
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 1:16 AM, Thibaut Despoulain
thib...@artillery.comwrote:
I've written a test for this here:
http://codeflow.org/issues/software-cursor.html
My observation from testing on linux is that I can't distinguish latency
for the software cursor from the OS cursor (or not by much anyway) in
google chrome. In firefox however there's noticable lag. Mileage may vary
for other platforms.
This is true, but sadly your test runs on an empty animation frame. If
your main thread is doing a lot of work already (barely hitting the 60fps
mark, as it is the case for demanding games), the lag will be much more
perceptible as you will most likely drop a frame every now and then.
I regard dropping below 60fps as an application defect for a realtime
interactive application where the users input is time critical.
*Reasons things drop below 60fps*
- You trigger GC-ing
- Your JS main thread takes up more than 16ms
- Your GPU processing time takes up more than 16ms
*Detecting the problem*
- You'd detect GC-ing by measuring the rate at which
requestAnimationFrame is fired. Gaps that appear periodically are typically
GC related.
- JS main thread time can be accurately measure with performance.now()
- GPU time will be measurable with EXT_disjoint_timer_query
http://www.khronos.org/registry/webgl/extensions/EXT_disjoint_timer_query/
*Solving the problem*
- Incremental GCs for JS will help. But they're not here yet, so one way
you can do something right now is to be very careful about the use of [],
{}, function(){}, new, DOM, createElement, innerHTML etc. It is possible to
eliminate GC caused frame drop almost entirely that way.
- Set yourself a time budget for the JS-main thread (that's well below
16ms) that you'll want to reach even on the lower-end of the expected
hardware. First try to keep everything below that budget in tests. If
something simply cannot be done in that time, split up into multi-frame
processing or shuffle it out to a webworker. Finally, if you still run into
the problem, keep constant tap on the JS time per frame, and when you run
over it reduce some processing (as in LOD, quality degradation etc.)
- EXT_disjoint_timer_query is not yet available, but once it does become
available you can use it to perform accurate testing on the lower end of
the hardware spectrum to identify rendering issues. And you can also use it
to dynamically react to pending performance issues by dialing down the LOD.
*Critical features for this use-case vendors have to offer, and soon*
- Showing the OS cursor, but containing it into a bounding region as
determined by a dom element (such as body, other elements)
- Implementing incremental GCs
- Implementing EXT_disjoint_timer_query
- Reducing input - output lag
I will point out that I have consistently called for fixes in these areas
during the last 2-3 years on the various discussions/specs/tickets on these
topics. So it's not surprising to me that these are issues now somebody
actually tries to make a real world, real time application in a browser.
I'm very glad that Artillery is doing the fine work of pushing this
boundary. I'd also like to ask the vendors to go the last mile as quickly
as feasible. Solving these issues is in the end, not just crucial for
realtime applications. Web applications get a lot of criticism for not
being able to compete in quality with native applications. This stems in
large part, from the fact that it is hard to make web applications as
responsive and snappy as native applications.
Re: [fullscreen] Problems with mouse-edge scrolling and games
The issue with pointerlock is that it requires the app to draw its own cursor instead of the OS cursor, which has at least two significant drawbacks: - Significant pointer lag between movement input and actual movement on screen compared to an OS-drawn cursor (yes, even at 60fps), - Inability to use DOM elements with mouse events for a game overlay/HUD.. We could work around #2, but #1 is a no-go for a twitch game such as a RTS where mouse accuracy and responsiveness is so critical. Something like a different kind of pointer lock that uses the OS cursor and only prevent the cursor from exiting the window canvas would be optimal. On Sat, Feb 22, 2014 at 2:39 PM, Florian Bösch pya...@gmail.com wrote: On Sat, Feb 22, 2014 at 11:22 PM, Florian Bösch pya...@gmail.com wrote: Caveat, I think Firefoxes implementaiton of the Pointerlock API might not follow the specification yet to make it possible to have pointerlock without fullscreen. Just checked this against a test I wrote a while ago http://codeflow.org/issues/pointerlock-test.html and consulted the pointerlock stats for IE. You can get pointerlock in both firefox and chrome without fullscreen (you can also get pointerlock and fullscreen simultaneously). IE seems to support pointerlock in versions that have webgl, but they seem to have added it only recently so support is still low (32%).
Re: [fullscreen] Problems with mouse-edge scrolling and games
I've written a test for this here: http://codeflow.org/issues/software-cursor.html My observation from testing on linux is that I can't distinguish latency for the software cursor from the OS cursor (or not by much anyway) in google chrome. In firefox however there's noticable lag. Mileage may vary for other platforms. This is true, but sadly your test runs on an empty animation frame. If your main thread is doing a lot of work already (barely hitting the 60fps mark, as it is the case for demanding games), the lag will be much more perceptible as you will most likely drop a frame every now and then. The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. Would that prevent vendors from implementing a pointer edge grabbing? (preventing the cursor to exit the window but keeping the OS cursor as-is while inside of it) On Sun, Feb 23, 2014 at 7:30 AM, Florian Bösch pya...@gmail.com wrote: On Sun, Feb 23, 2014 at 4:18 PM, Brandon Jones bajo...@google.com wrote: - it's possible to theme the OS cursor using custom images with CSS. https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url Although that doesn't absolve vendors from fixing the latency issue even if native pointers where to be made available during pointerlock. The reason is that cursors come in more flavors than an image. For example they could come in some variety of 3D rendered representations useful for the game in question. - The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. Isn't that only a concern if you want to capture the cursor, not when you display the OS cursor? - You already mentioned some issues with synthetic mouse events, but unfortunately it's worse than you suspect. For example: sending a synthetic click event to a checkbox doesn't actually change it's checked state. (Not the last time I tried anyway) select controls also have a hard time with synthetic events, and there's a whole host of other sub rely broken things. :( Is there a motivation not to make it work? Clickjacking?
Re: [fullscreen] Problems with mouse-edge scrolling and games
Also: At the moment we're using CSS cursors to give visual feedback on in-game hover/action states which works pretty well and is a breeze to implement. On Sun, Feb 23, 2014 at 4:16 PM, Thibaut Despoulain thib...@artillery.comwrote: I've written a test for this here: http://codeflow.org/issues/software-cursor.html My observation from testing on linux is that I can't distinguish latency for the software cursor from the OS cursor (or not by much anyway) in google chrome. In firefox however there's noticable lag. Mileage may vary for other platforms. This is true, but sadly your test runs on an empty animation frame. If your main thread is doing a lot of work already (barely hitting the 60fps mark, as it is the case for demanding games), the lag will be much more perceptible as you will most likely drop a frame every now and then. The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. Would that prevent vendors from implementing a pointer edge grabbing? (preventing the cursor to exit the window but keeping the OS cursor as-is while inside of it) On Sun, Feb 23, 2014 at 7:30 AM, Florian Bösch pya...@gmail.com wrote: On Sun, Feb 23, 2014 at 4:18 PM, Brandon Jones bajo...@google.comwrote: - it's possible to theme the OS cursor using custom images with CSS. https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url Although that doesn't absolve vendors from fixing the latency issue even if native pointers where to be made available during pointerlock. The reason is that cursors come in more flavors than an image. For example they could come in some variety of 3D rendered representations useful for the game in question. - The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. Isn't that only a concern if you want to capture the cursor, not when you display the OS cursor? - You already mentioned some issues with synthetic mouse events, but unfortunately it's worse than you suspect. For example: sending a synthetic click event to a checkbox doesn't actually change it's checked state. (Not the last time I tried anyway) select controls also have a hard time with synthetic events, and there's a whole host of other sub rely broken things. :( Is there a motivation not to make it work? Clickjacking?
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 4:17 AM, Florian Bösch pya...@gmail.com wrote: On Mon, Feb 24, 2014 at 1:16 AM, Thibaut Despoulain thib...@artillery.com wrote: I've written a test for this here: http://codeflow.org/issues/software-cursor.html My observation from testing on linux is that I can't distinguish latency for the software cursor from the OS cursor (or not by much anyway) in google chrome. In firefox however there's noticable lag. Mileage may vary for other platforms. This is true, but sadly your test runs on an empty animation frame. If your main thread is doing a lot of work already (barely hitting the 60fps mark, as it is the case for demanding games), the lag will be much more perceptible as you will most likely drop a frame every now and then. I regard dropping below 60fps as an application defect for a realtime interactive application where the users input is time critical. First, mouse input is always time-critical, even if the application itself is not. Even for a spreadsheet, the mouse must still move responsively, since pointing devices are difficult to use with any delay. Some games only run graphics at 30 FPS (like it or not), but the pointer if any must still update at 60 FPS. Second, system-driven pointers allow better responsiveness than is possible at the application level. They can process mouse input and update the mouse cursor right when the backbuffer is flipped, giving the lowest possible latency. Applications (especially web apps) simply can't do that. Going forward this could be even more severe. For example, AirPlay video streaming has significant video delay, but they could send mouse movement over a sideband and draw it on the rendering device, allowing responsive mouse movement where it would otherwise be impossible. It's not the application's job to keep the mouse cursor responsive, it's the system's. Hiding the system mouse cursor and drawing one manually is always a bad idea. -- Glenn Maynard
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 5:18 PM, Glenn Maynard gl...@zewt.org wrote: It's not the application's job to keep the mouse cursor responsive, it's the system's. Hiding the system mouse cursor and drawing one manually is always a bad idea. That's a wonderful argument. And now we look at an FPS game, or an Oculus Rift title, or something that requires something else than a picture cursor like say, an on terrain selection, a bounding box selection, a 3D ray picking selection/cursor, or anything like that. Always a bad idea, sure. How about you think about that again hm?
Re: [fullscreen] Problems with mouse-edge scrolling and games
(More reasons: it's very likely that you'll end up implementing a cursor with different motion and acceleration, a different feel, than the real mouse cursor. It also breaks accessibility features, like mouse trails.) On Mon, Feb 24, 2014 at 10:30 AM, Florian Bösch pya...@gmail.com wrote: On Mon, Feb 24, 2014 at 5:18 PM, Glenn Maynard gl...@zewt.org wrote: It's not the application's job to keep the mouse cursor responsive, it's the system's. Hiding the system mouse cursor and drawing one manually is always a bad idea. That's a wonderful argument. And now we look at an FPS game, or an Oculus Rift title, or something that requires something else than a picture cursor like say, an on terrain selection, a bounding box selection, a 3D ray picking selection/cursor, or anything like that. Always a bad idea, sure. How about you think about that again hm? This doesn't seem to relate to the discussion, which is about mouse pointers. -- Glenn Maynard
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 5:40 PM, Glenn Maynard gl...@zewt.org wrote: (More reasons: it's very likely that you'll end up implementing a cursor with different motion and acceleration, a different feel, than the real mouse cursor. It also breaks accessibility features, like mouse trails.) Oh I agree, if your usecase fits a mouse cursor of the style that the OS offers, it's definitely preferable to have the OS mouse cursor. And I distinctly remember arguing prior to the pointerlock specification that it would be immensely useful to have the ability to show the cursor, but capture the pointer inside an area. And I was pointed, at the time, to the solution to draw a software mouse cursor... So now what we're having this discussion, I find appropriate to have pointed out as a possibility (to draw the software cursor), which met some, let's call them difficulties. Now, I also just made a nice list which mentions 4 things to do, among them, add this ability. I consider this now settled, because I think we all agree on this. We might just need to agree on the limitations as they partain to relative mouse movement reporting when you're showing the OS cursor. And I think that's relatively easy to agree on, you can't rely on relative motion outside of the constrained area if you show the OS cursor. This doesn't seem to relate to the discussion, which is about mouse pointers. It is about input - output, essentially. Intput as it comes from your pointing device, and output, as it reflects in your application. The OS mouse cursor is but one example of such reflection. There are many more usecases that would like to benefit from a low input - output latency, one example of which is the variety of software cursors which may take the shape of a picture, or something else completely. But moreover there are usecases such as viewing controls that have the exact same need, notably FPS shooters, for instance, or Oculus rift titles and so forth. So, it matters a great deal to the, because you can alleviate some of the issues while simultaneously solving a whole host of other issues.
Re: [fullscreen] Problems with mouse-edge scrolling and games
Glenn Maynard wrote: It's not the application's job to keep the mouse cursor responsive, it's the system's. Hiding the system mouse cursor and drawing one manually is always a bad idea. Agreed! In the same vein, programmers cannot avoid GC pauses without relying on pause-free or at least incremental GC (which BTW some browsers' JS engines have already, e.g., SpiderMonkey in Firefox), or as a real alternative, cross-compiling C or C++ for example via Emscripten, to allocate heap memory from a typed array. Florian, your goals are good, but the means to those ends must involve better runtimes or compilers -- not on JS programmers working harder to avoid GC while still somehow allocating objects frequently and even implicitly. /be
Re: [fullscreen] Problems with mouse-edge scrolling and games
Thank you Ian for the clear use case write up (and to Artillery for progress on a great product), Florian Brandon and Glenn for discussion contributions. As Florian mentioned, we did discuss this use case in 2011 and at that time I attempted to summarize the situation in the pointer lock spec FAQ [1]. Please do read that discussion of the related 3 topics of clipping, motion while clipped, and raw data. One way to move forward on those topics is to prototype functional solutions on Windows, Mac, and Linux and illuminate a practical solution that browsers may adopt. I did an investigation in 2011 and did not find a solution. For clipping only I believe Mac was the only platform I didn't find a solution for. Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we can implement the functionality, we can discuss how to express this in an API. [1] https://dvcs.w3.org/hg/pointerlock/raw-file/default/index.html#why-bundle-all-functionality-hiding-cursor-providing-mouse-deltas-instead-of-using-css-to-hide-the-cursor-always-providing-delta-values-and-offering-an-api-to-restrict-the-cursor-movement-to-a-portion-of-the-web-page
Re: [fullscreen] Problems with mouse-edge scrolling and games
@Florian: That is very well put. Trust me, we are well aware of time budgets and GC pauses. The thing is that even when planning frame budgets and object allocations ahead, you still have external parameters you cannot really control (ie. Another tab running flash or some other intensive task, another software running in the background, etc.). This is why an OS-drawn cursor is so critical (in our use case that is, since we rely on a simple 2d cursor styled with CSS and not a 3d one) Furthermore, when targeting low-end hardware, a 30fps render loop is enough for a decent graphical experience, while being completely mediocre and unplayable on the input side. As for your last point, Critical features for this use-case vendors have to offer, and soon”, we completely agree and would love to see those features implemented sooner than later. -- WebGL guru @Artillery Mail: thib...@artillery.com Tweet: @BKcore From: Brendan Eich Brendan Eich Reply: Brendan Eich bren...@secure.meer.net Date: February 24, 2014 at 9:47:24 AM To: Glenn Maynard gl...@zewt.org Subject: Re: [fullscreen] Problems with mouse-edge scrolling and games Glenn Maynard wrote: It's not the application's job to keep the mouse cursor responsive, it's the system's. Hiding the system mouse cursor and drawing one manually is always a bad idea. Agreed! In the same vein, programmers cannot avoid GC pauses without relying on pause-free or at least incremental GC (which BTW some browsers' JS engines have already, e.g., SpiderMonkey in Firefox), or as a real alternative, cross-compiling C or C++ for example via Emscripten, to allocate heap memory from a typed array. Florian, your goals are good, but the means to those ends must involve better runtimes or compilers -- not on JS programmers working harder to avoid GC while still somehow allocating objects frequently and even implicitly. /be
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 6:47 PM, Brendan Eich bren...@secure.meer.netwrote: Glenn Maynard wrote: It's not the application's job to keep the mouse cursor responsive, it's the system's. Hiding the system mouse cursor and drawing one manually is always a bad idea. Agreed! Like I say, some usecases are fine with OS cursors. But that doesn't mean that somehow, vendors are absolved from improving input - output latency issues even if pointerlock is updated to allow OS cursor showing, which I'm all for. There are a lot of usecases that involve pointing devices, and pointing metaphors, or view controls, virtual helmets, and so forth, that cannot properly function with a high input - output latency. For this reason it's imperative not only to address the ability to make the OS cursor visible, but also to continue working on low latency input - output. In the same vein, programmers cannot avoid GC pauses without relying on pause-free or at least incremental GC (which BTW some browsers' JS engines have already, e.g., SpiderMonkey in Firefox), or as a real alternative, cross-compiling C or C++ for example via Emscripten, to allocate heap memory from a typed array. Florian, your goals are good, but the means to those ends must involve better runtimes or compilers -- not on JS programmers working harder to avoid GC while still somehow allocating objects frequently and even implicitly. I agree that things aren't today how they should be for realtime applications with GCs. And it's true that GCs are getting better. But, it is the status of today, that a JS programmer has to work harder to make a glitch/stutter/jerk free realtime applications. A better GC can improve this situation. However, that doesn't mean that you can forget about GCing and frame budgets. A realtime programmer will always have to be conscious not to overload the GC even if it's incremental. Because when the incremental GC would not manage to get rid of the garbage faster than it is produced, it would have to resort to more drastic pauses to rectify that situation. Fortunately the act of being GC conscious for a non incremental GC, and for an incremental one, is very similar. You try to avoid triggering it. So in that, you can start that work today, it will not be in vain once GCs get better in some far flung future.
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 7:07 PM, Vincent Scheib sch...@google.com wrote: Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we can implement the functionality, we can discuss how to express this in an API. Would using Quarz CGWarpMouseCursorPosition work where you'd clamp the passed position into the desired rectangle? https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html#//apple_ref/c/func/CGWarpMouseCursorPosition
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 10:37 AM, Florian Bösch pya...@gmail.com wrote: On Mon, Feb 24, 2014 at 7:07 PM, Vincent Scheib sch...@google.com wrote: Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we can implement the functionality, we can discuss how to express this in an API. Would using Quarz CGWarpMouseCursorPosition work where you'd clamp the passed position into the desired rectangle? https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html#//apple_ref/c/func/CGWarpMouseCursorPosition I believe no, because it would allow the pointer to escape the region before being warped back, permitting escape if clicked at that time as well.
Re: [fullscreen] Problems with mouse-edge scrolling and games
Right, so you'd CGAssociateMouseAndCursorPosition(false) and then use CGWarpMouseCursorPosition or CGDisplayMouseCursorToPoint to move it where you want it to be, but clamped inside the rect. As long as you keep pumping the event loop separately that does this (as fast as possible) it shouldn't be perceptively different from an OS cursor. On Mon, Feb 24, 2014 at 7:41 PM, Vincent Scheib sch...@google.com wrote: On Mon, Feb 24, 2014 at 10:37 AM, Florian Bösch pya...@gmail.com wrote: On Mon, Feb 24, 2014 at 7:07 PM, Vincent Scheib sch...@google.comwrote: Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we can implement the functionality, we can discuss how to express this in an API. Would using Quarz CGWarpMouseCursorPosition work where you'd clamp the passed position into the desired rectangle? https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html#//apple_ref/c/func/CGWarpMouseCursorPosition I believe no, because it would allow the pointer to escape the region before being warped back, permitting escape if clicked at that time as well.
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 12:19 PM, Florian Bösch pya...@gmail.com wrote: Like I say, some usecases are fine with OS cursors. But that doesn't mean that somehow, vendors are absolved from improving input - output latency issues even if pointerlock is updated to allow OS cursor showing, which I'm all for. Only if low latency is important to the application. It's very important for some games (rhythm games), moderately important for some (first-person games controlled with a mouse, which in turn is more sensitive than with a gamepad), and not very important at all for others (turn-based strategy). How much time to spend on this is a business decision, since this can be very time-consuming and can require visual tradeoffs. There are a lot of usecases that involve pointing devices, and pointing metaphors, or view controls, virtual helmets, and so forth, that cannot properly function with a high input - output latency. For this reason it's imperative not only to address the ability to make the OS cursor visible, but also to continue working on low latency input - output. True, but tangental. :) (My last job was making music rhythm games, which are more sensitive to consistent framerates than any other game genre I know of--one dropped frame can wreck someone's game--so I'm sympathetic to the cause.) On Fri, Feb 21, 2014 at 7:00 PM, Ian Langworth i...@artillery.com wrote: The better option is to go fullscreen with the Fullscreen API, but this has problems as well. There are certain behaviors that occur when moving the cursor to the top and bottom of the screen, I think that going fullscreen is the right approach, since locking the mouse into the window while not fullscreen is really weird and rare, at least in Windows. By going fullscreen, this hooks into the same UI design to allow the user to escape. Even if this was supported in a window, there'd still have to be some UI to tell the user how to exit, which could end up having the same problem. I've been annoyed by the edge-of-screen browser behavior too. It's a part of the screen where you might want to put something, like navigation UI. I haven't come up with a better solution, though. I don't think having a stronger fullscreen mode that asks the user for more permission will fly. Browsers try very hard to avoid asking for special permissions--people will just agree without reading it, then won't know how to escape from the app. I think that for your use case of edge scrolling, having a fullscreen notice appear at the top is OK (if a little ugly), as long as it's transparent to mouse events so you can still see the mouse moving around (or else you might see the mouse move to 20 pixels from the top, then never see it actually reach the top, so you'd never start scrolling). Menus and address bars appearing seems like a bug. That makes sense for the fullscreen you get by hitting F11 in Windows or Command-Shift-F in OSX, but application fullscreen should just act like a game, and keep as much as possible out of the way. -- Glenn Maynard
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 8:21 PM, Glenn Maynard gl...@zewt.org wrote: I think that going fullscreen is the right approach, since locking the mouse into the window while not fullscreen is really weird and rare, at least in Windows. It's quite common for games to have a cursor, grab the pointer and not be fullscreen. Of course most games that allow for this, use software cursors, and are apparently not having much problems with it. By going fullscreen, this hooks into the same UI design to allow the user to escape. Even if this was supported in a window, there'd still have to be some UI to tell the user how to exit, which could end up having the same problem. A sidenote, if you have more than one monitor, going fullscreen will not lock the pointer on screen, obviously. It's not terribly common perhaps to have more than one monitor, but it's also not that rare. I've been annoyed by the edge-of-screen browser behavior too. It's a part of the screen where you might want to put something, like navigation UI. I haven't come up with a better solution, though. I don't think having a stronger fullscreen mode that asks the user for more permission will fly. Browsers try very hard to avoid asking for special permissions--people will just agree without reading it, then won't know how to escape from the app. I think that for your use case of edge scrolling, having a fullscreen notice appear at the top is OK (if a little ugly), as long as it's transparent to mouse events so you can still see the mouse moving around (or else you might see the mouse move to 20 pixels from the top, then never see it actually reach the top, so you'd never start scrolling). Menus and address bars appearing seems like a bug. That makes sense for the fullscreen you get by hitting F11 in Windows or Command-Shift-F in OSX, but application fullscreen should just act like a game, and keep as much as possible out of the way. For a fast paced game where you might click and select and do whatnot, having a slidedown from the top of the window when you hit the border is not acceptable. People will click it accidentally a lot for instance when doing a selection. Not being able to offer a fullscreen button in the game is also bad UX. You end up with explanations for the user to Please press F11 to get into fullscreen. You should never have to explain to a user what ritual he has to perform, if instead you can trigger that action without the ritual.
Re: [fullscreen] Problems with mouse-edge scrolling and games
Exactly. Having an “exit from fullscreen” bar or some OS auto-hide elements like docks and task bars popping up at the edges when you play an intense twitch game is simply horrendous. We have several playtests feedbacks arguing in that direction. Not to mention that a lot of our alpha players have multi-monitors configurations where the mouse is “escaping” on the secondary monitor, even while full screened, which is a terrible experience. -- WebGL guru @Artillery Mail: thib...@artillery.com Tweet: @BKcore From: Florian Bösch Florian Bösch Reply: Florian Bösch pya...@gmail.com Date: February 24, 2014 at 11:32:13 AM To: Glenn Maynard gl...@zewt.org Subject: Re: [fullscreen] Problems with mouse-edge scrolling and games On Mon, Feb 24, 2014 at 8:21 PM, Glenn Maynard gl...@zewt.org wrote: I think that going fullscreen is the right approach, since locking the mouse into the window while not fullscreen is really weird and rare, at least in Windows. It's quite common for games to have a cursor, grab the pointer and not be fullscreen. Of course most games that allow for this, use software cursors, and are apparently not having much problems with it. By going fullscreen, this hooks into the same UI design to allow the user to escape. Even if this was supported in a window, there'd still have to be some UI to tell the user how to exit, which could end up having the same problem. A sidenote, if you have more than one monitor, going fullscreen will not lock the pointer on screen, obviously. It's not terribly common perhaps to have more than one monitor, but it's also not that rare. I've been annoyed by the edge-of-screen browser behavior too. It's a part of the screen where you might want to put something, like navigation UI. I haven't come up with a better solution, though. I don't think having a stronger fullscreen mode that asks the user for more permission will fly. Browsers try very hard to avoid asking for special permissions--people will just agree without reading it, then won't know how to escape from the app. I think that for your use case of edge scrolling, having a fullscreen notice appear at the top is OK (if a little ugly), as long as it's transparent to mouse events so you can still see the mouse moving around (or else you might see the mouse move to 20 pixels from the top, then never see it actually reach the top, so you'd never start scrolling). Menus and address bars appearing seems like a bug. That makes sense for the fullscreen you get by hitting F11 in Windows or Command-Shift-F in OSX, but application fullscreen should just act like a game, and keep as much as possible out of the way. For a fast paced game where you might click and select and do whatnot, having a slidedown from the top of the window when you hit the border is not acceptable. People will click it accidentally a lot for instance when doing a selection. Not being able to offer a fullscreen button in the game is also bad UX. You end up with explanations for the user to Please press F11 to get into fullscreen. You should never have to explain to a user what ritual he has to perform, if instead you can trigger that action without the ritual.
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Sun, Feb 23, 2014 at 1:57 AM, Thibaut Despoulain thib...@artillery.com wrote: The issue with pointerlock is that it requires the app to draw its own cursor instead of the OS cursor I fully agree with motivation, it is usually preferrable to give the user an OS-themed cursor (not always, but often). - Significant pointer lag between movement input and actual movement on screen compared to an OS-drawn cursor (yes, even at 60fps), I've written a test for this here: http://codeflow.org/issues/software-cursor.html My observation from testing on linux is that I can't distinguish latency for the software cursor from the OS cursor (or not by much anyway) in google chrome. In firefox however there's noticable lag. Mileage may vary for other platforms. - Inability to use DOM elements with mouse events for a game overlay/HUD. The test I've written here http://codeflow.org/issues/software-cursor.htmlalso tests mouse event synthesis (as hinted at by an example in the pointerlock spec) and it works satisfactory in chrome and firefox on linux. To get all subtleties right one would also have to do most other mouse events I guess (mouseover, mouseout, mousemove etc.). On Sun, Feb 23, 2014 at 2:15 AM, Brandon Jones bajo...@google.com wrote: Chrome a software cursor will experience visible lag, which these types of games are highly sensitive to. Not as much lag as I expected, see this test: http://codeflow.org/issues/software-cursor.html I'm not sure about the latency of Firefox or others Pretty bad, see this test: http://codeflow.org/issues/software-cursor.html , but in general a hardware cursor is preferable any time you have a visible cursor. As a result this is still a use case worth considering. It's not always preferrable (sometimes people want custom cursors). But in general it is preferrable. When providing the OS cursor there's several issues to be addressed: - If you are in fullscreen, capture the cursor to the viewport but show the OS cursor, that does *NOT* mean you'll want the annoying slide-down that some browsers provide if you hit the top edge. If you wanted that, you'd never have restricted the cursor in the first place. I would suggest as a resolution that if somebody requests cursor restriction, and gets it, that the slide down is never shown regardless if the OS cursor is shown or not. - It should be possible to show and hide the OS cursor separately, but retain pointerlock. The reason for that is that you might want to avoid synthesizing your own cursor, but you may also want to keep pointerlock because your game switched to some UI interaction for instance from an FPS interaction mode. I would suggest this could be resolved with an additional API call of the form: document.hideCursor() and document.showCursor() or something similar. If the sequence is myContainer.requestPointerLock(); document.showCursor(); the cursor is shown, but confined to the container. Outside of being in pointerlock, these calls would have no effect.
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Sun, Feb 23, 2014 at 9:55 AM, Florian Bösch pya...@gmail.com wrote: - Inability to use DOM elements with mouse events for a game overlay/HUD. The test I've written here http://codeflow.org/issues/software-cursor.html also tests mouse event synthesis (as hinted at by an example in the pointerlock spec) and it works satisfactory in chrome and firefox on linux. To get all subtleties right one would also have to do most other mouse events I guess (mouseover, mouseout, mousemove etc.). I did some more testing on this. Synthetic mouse events are limited in that they cannot trigger some state like hover styles. Other things like focus/blur require special handling. It's also left to the synthetic event creator to synthesize compound events like over/out/enter/leave and their semantic as supplying the primitive mouse events (click, mousemove) doesn't automatically synthesize the others.
Re: [fullscreen] Problems with mouse-edge scrolling and games
Several points to add: - it's possible to theme the OS cursor using custom images with CSS. https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url - The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. - You already mentioned some issues with synthetic mouse events, but unfortunately it's worse than you suspect. For example: sending a synthetic click event to a checkbox doesn't actually change it's checked state. (Not the last time I tried anyway) select controls also have a hard time with synthetic events, and there's a whole host of other sub rely broken things. :( Thanks for the tests! On Sunday, February 23, 2014, Florian Bösch pya...@gmail.com wrote: On Sun, Feb 23, 2014 at 9:55 AM, Florian Bösch pya...@gmail.comjavascript:_e(%7B%7D,'cvml','pya...@gmail.com'); wrote: - Inability to use DOM elements with mouse events for a game overlay/HUD. The test I've written here http://codeflow.org/issues/software-cursor.html also tests mouse event synthesis (as hinted at by an example in the pointerlock spec) and it works satisfactory in chrome and firefox on linux. To get all subtleties right one would also have to do most other mouse events I guess (mouseover, mouseout, mousemove etc.). I did some more testing on this. Synthetic mouse events are limited in that they cannot trigger some state like hover styles. Other things like focus/blur require special handling. It's also left to the synthetic event creator to synthesize compound events like over/out/enter/leave and their semantic as supplying the primitive mouse events (click, mousemove) doesn't automatically synthesize the others.
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Sun, Feb 23, 2014 at 4:18 PM, Brandon Jones bajo...@google.com wrote: - it's possible to theme the OS cursor using custom images with CSS. https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url Although that doesn't absolve vendors from fixing the latency issue even if native pointers where to be made available during pointerlock. The reason is that cursors come in more flavors than an image. For example they could come in some variety of 3D rendered representations useful for the game in question. - The reason the cursor is hidden when the pointer is locked is that some OSes don't have the ability to report relative mouse movement correctly at screen edges. This requires the cursors to constantly be reset to the center of the screen, which obviously would look strange if the cursor was visible. Isn't that only a concern if you want to capture the cursor, not when you display the OS cursor? - You already mentioned some issues with synthetic mouse events, but unfortunately it's worse than you suspect. For example: sending a synthetic click event to a checkbox doesn't actually change it's checked state. (Not the last time I tried anyway) select controls also have a hard time with synthetic events, and there's a whole host of other sub rely broken things. :( Is there a motivation not to make it work? Clickjacking?
Re: [fullscreen] Problems with mouse-edge scrolling and games
On Sat, Feb 22, 2014 at 11:22 PM, Florian Bösch pya...@gmail.com wrote: Caveat, I think Firefoxes implementaiton of the Pointerlock API might not follow the specification yet to make it possible to have pointerlock without fullscreen. Just checked this against a test I wrote a while ago http://codeflow.org/issues/pointerlock-test.html and consulted the pointerlock stats for IE. You can get pointerlock in both firefox and chrome without fullscreen (you can also get pointerlock and fullscreen simultaneously). IE seems to support pointerlock in versions that have webgl, but they seem to have added it only recently so support is still low (32%).
Re: [fullscreen] Problems with mouse-edge scrolling and games
Pointerlock should solve these problems in the following fashion: - When the user clicks into the app, request pointerlock - Use it to give him a cursor drawn by you - That way you can keep the interaction inside your game and accurately detect borders etc. I run http://webglstats.com/ and It also records the availability of pointerlock. This is relative to everybody who has webgl and it currently stands at 94.1%. Caveat, I think Firefoxes implementaiton of the Pointerlock API might not follow the specification yet to make it possible to have pointerlock without fullscreen. I know that it's possible to get pointerlock in google chrome without fullscreen. Links: - http://www.html5rocks.com/en/tutorials/pointerlock/intro/ - http://www.w3.org/TR/pointerlock/ On Sat, Feb 22, 2014 at 2:00 AM, Ian Langworth i...@artillery.com wrote: Hi everyone, We're building a console- and native-quality game in the browser using JavaScript and WebGL. You can see a very early version of the game in this video: https://youtu.be/NiCy5igO9-I . It's a realtime strategy (RTS) game, like StarCraft[1] or Command Conquer[2], and moving the cursor to the edge of the screen is the primary way that users move around the map. We have a single requirement: Moving the cursor to, and possibly past, the edge of the window or screen should pan the camera in that direction. In windowed mode there are a couple of problems: Players might click outside the window accidentally, pixels outside the window are wasted screen real estate, mousemove events aren't fired when the cursor is outside the window, and sometimes the cursor moves so quickly that it misses the area of the window where we do edge detection. The better option is to go fullscreen with the Fullscreen API, but this has problems as well. There are certain behaviors that occur when moving the cursor to the top and bottom of the screen, which I've illustrated in the following videos using a quick edge-scrolling demo[3].. (Please ignore the problems with cursor detection in the Windows videos. It only occurred while the screen recording software was active.): Chrome, Windows 7 https://www.dropbox.com/s/t9kyo8s5am76ezg/chrome%20win7%20edges.avi The fullscreen notice appears every time the cursor is moved to the top. Chrome, Mac OS X https://www.dropbox.com/s/ke1mr5te2dwgvor/chrome%20osx%20edges.mov A poor experience with the menu and address bars appearing at the top and Dock appearing at the bottom. Firefox, Windows 7 https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges.avihttps://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges..avi No problems with the fullscreen experience! Firefox, Mac OS X https://www.dropbox.com/s/0bkdx71ir0yhw88/firefox%20osx%20edges.mov Menu bar appears at top, along with some mysterious window chrome, and the Dock appears at the bottom as well. These behaviors are to remind the user about fullscreen and lessen the chance of phishing. But, as you might imagine, they are troublesome when you're in the heat of battle and are trying to crush your opponent. I would like to solicit the list for opinions on how we can improve the fullscreen UI for games and give players the best experience possible. Some options that come to mind: (A) Provide an alternate fullscreen API or option which provides a more complete experience, but with more dire warnings to the user. For example, pass in a flag to requestFullscreen() which gives the page full control over the screen (no fullscreen reminder notifications, menu bars, or Docks) and even the keyboard (so games can use the Escape key), but entering this mode presents a much more intense warning to the user like the invalid certificate warnings. (B) Maintain the current API, but put an option in the preferences or flags which makes the browser's fullscreen mode more complete and without interruption. Hardcore gamers will likely accept this extra step if it means getting an optimal experience. (C) Propose a new Mouse-Edge Detection API, which might solve these problems and provide games with better cursor detection overall. I appreciate any and all feedback. [1] http://youtu.be/Qb0VzbxdP4U?t=10m14s [2] http://youtu.be/l41hG-fVDN4?t=3m34s [3] http://jsfiddle.net/statico/F8sjw/show/
Re: [fullscreen] Problems with mouse-edge scrolling and games
Pointer lock does indeed provide a workaround for this, but it's worth noting that with the current architecture of Chrome a software cursor will experience visible lag, which these types of games are highly sensitive to. I'm not sure about the latency of Firefox or others, but in general a hardware cursor is preferable any time you have a visible cursor. As a result this is still a use case worth considering. --Brandon On Saturday, February 22, 2014, Florian Bösch pya...@gmail.com wrote: Pointerlock should solve these problems in the following fashion: - When the user clicks into the app, request pointerlock - Use it to give him a cursor drawn by you - That way you can keep the interaction inside your game and accurately detect borders etc. I run http://webglstats.com/ and It also records the availability of pointerlock. This is relative to everybody who has webgl and it currently stands at 94.1%. Caveat, I think Firefoxes implementaiton of the Pointerlock API might not follow the specification yet to make it possible to have pointerlock without fullscreen. I know that it's possible to get pointerlock in google chrome without fullscreen. Links: - http://www.html5rocks.com/en/tutorials/pointerlock/intro/ - http://www.w3.org/TR/pointerlock/ On Sat, Feb 22, 2014 at 2:00 AM, Ian Langworth i...@artillery.comjavascript:_e(%7B%7D,'cvml','i...@artillery.com'); wrote: Hi everyone, We're building a console- and native-quality game in the browser using JavaScript and WebGL. You can see a very early version of the game in this video: https://youtu.be/NiCy5igO9-I . It's a realtime strategy (RTS) game, like StarCraft[1] or Command Conquer[2], and moving the cursor to the edge of the screen is the primary way that users move around the map. We have a single requirement: Moving the cursor to, and possibly past, the edge of the window or screen should pan the camera in that direction. In windowed mode there are a couple of problems: Players might click outside the window accidentally, pixels outside the window are wasted screen real estate, mousemove events aren't fired when the cursor is outside the window, and sometimes the cursor moves so quickly that it misses the area of the window where we do edge detection. The better option is to go fullscreen with the Fullscreen API, but this has problems as well. There are certain behaviors that occur when moving the cursor to the top and bottom of the screen, which I've illustrated in the following videos using a quick edge-scrolling demo[3].. (Please ignore the problems with cursor detection in the Windows videos. It only occurred while the screen recording software was active.): Chrome, Windows 7 https://www.dropbox.com/s/t9kyo8s5am76ezg/chrome%20win7%20edges.avi The fullscreen notice appears every time the cursor is moved to the top. Chrome, Mac OS X https://www.dropbox.com/s/ke1mr5te2dwgvor/chrome%20osx%20edges.mov A poor experience with the menu and address bars appearing at the top and Dock appearing at the bottom. Firefox, Windows 7 https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges.avihttps://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges..avi No problems with the fullscreen experience! Firefox, Mac OS X https://www.dropbox.com/s/0bkdx71ir0yhw88/firefox%20osx%20edges.mov Menu bar appears at top, along with some mysterious window chrome, and the Dock appears at the bottom as well. These behaviors are to remind the user about fullscreen and lessen the chance of phishing. But, as you might imagine, they are troublesome when you're in the heat of battle and are trying to crush your opponent. I would like to solicit the list for opinions on how we can improve the fullscreen UI for games and give players the best experience possible. Some options that come to mind: (A) Provide an alternate fullscreen API or option which provides a more complete experience, but with more dire warnings to the user. For example, pass in a flag to requestFullscreen() which gives the page full control over the screen (no fullscreen reminder notifications, menu bars, or Docks) and even the keyboard (so games can use the Escape key), but entering this mode presents a much more intense warning to the user like the invalid certificate warnings. (B) Maintain the current API, but put an option in the preferences or flags which makes the browser's fullscreen mode more complete and without interruption. Hardcore gamers will likely accept this extra step if it means getting an optimal experience. (C) Propose a new Mouse-Edge Detection API, which might solve these problems and provide games with better cursor detection overall. I appreciate any and all feedback. [1] http://youtu.be/Qb0VzbxdP4U?t=10m14s [2] http://youtu.be/l41hG-fVDN4?t=3m34s [3] http://jsfiddle.net/statico/F8sjw/show/
