[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi Hironori, Upon further reflection, I'm concerned about some of the fixes around these issues interacting badly with the Cocoa restructuring work I'm doing. Could you hold off on the OSX side of the changes till the end of the month to give me time to transition the Mac infrastructure to work better with Cocoa? Best regards, Jeremy 2009/6/7 Jeremy Moskovich jer...@chromium.org Hi Hironori, Thanks for the detailed report. I haven't nearly the same insight into the text input system as you and Avi do, but from what I've seen it may be possible for us to do at least a marginally better job using the built-in system routines and thus avoiding many sticky bugs e.g. http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts should be fixed by a CL I'm working on at the moment. The solution is to go through the native Cocoa methods to have the menu item fire the appropriate objc selector on the RHWV rather than trying to process the raw keystroke in the renderer process. I also think we can get Cocoa to translate emacs editing commands (such as cntrl-a) into the appropriate WebKit editor commands for us. Best regards, Jeremy 2009/6/5 Hironori Bono (坊野 博典) hb...@chromium.org Hi Jeremy, et al, Sorry for my slow update. This is the draft report of my investigation that implements the NSTextInput protocol on Mac, implements the GtkIMContext on Linux and observes keyboard events on Windows, Mac, and Linux. (Sorry, I would like to send it as a PDF file because the google site somehow compresses its figures too much to make its characters unreadable.) Since I'm still finding the best solutions for the issues, any comments and suggestions are definitely welcome. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/5/29 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi Jeremy, It is not a problem at all. Thank you for notifying this. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/6/9 Jeremy Moskovich jer...@chromium.org: Hi Hironori, Upon further reflection, I'm concerned about some of the fixes around these issues interacting badly with the Cocoa restructuring work I'm doing. Could you hold off on the OSX side of the changes till the end of the month to give me time to transition the Mac infrastructure to work better with Cocoa? Best regards, Jeremy 2009/6/7 Jeremy Moskovich jer...@chromium.org Hi Hironori, Thanks for the detailed report. I haven't nearly the same insight into the text input system as you and Avi do, but from what I've seen it may be possible for us to do at least a marginally better job using the built-in system routines and thus avoiding many sticky bugs e.g. http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts should be fixed by a CL I'm working on at the moment. The solution is to go through the native Cocoa methods to have the menu item fire the appropriate objc selector on the RHWV rather than trying to process the raw keystroke in the renderer process. I also think we can get Cocoa to translate emacs editing commands (such as cntrl-a) into the appropriate WebKit editor commands for us. Best regards, Jeremy 2009/6/5 Hironori Bono (坊野 博典) hb...@chromium.org Hi Jeremy, et al, Sorry for my slow update. This is the draft report of my investigation that implements the NSTextInput protocol on Mac, implements the GtkIMContext on Linux and observes keyboard events on Windows, Mac, and Linux. (Sorry, I would like to send it as a PDF file because the google site somehow compresses its figures too much to make its characters unreadable.) Since I'm still finding the best solutions for the issues, any comments and suggestions are definitely welcome. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/5/29 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi Nico, Thank you for your comment and sorry for confusing you. 2009/6/6 Nico Weber tha...@chromium.org: Hi, re Type ctrl-a on hebrew keyboard: On OS X, ctrl-a usually does not select all text (that's what cmd-a does) but goes to the beginning of the current line, as in emacs. But that can be overridden ( http://www.hcs.harvard.edu/~jrus/site/cocoa-text.html ). Yes, we noticed this control+a key should move the input cursor at the beginning of a line on Mac. Nevertheless, this issue happens also on US keyboards. The purpose of this discussion is: how we should handle non-US keyboards as well as US keyboards, so, the expected behavior for this control+a case is the one when we type the control+a keys on a US keyboard even though this is not expected behavior of Mac OS X. Mac people, Is it possible to give me whether or not there are any bugs filed to our buganizer for this shortcut-incompatibility issue? Regards, Hironori Bono E-mail: hb...@chromium.org --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi James Su, Thank you for your comments. On Sat, Jun 6, 2009 at 1:17 AM, Zhe Sujames...@gmail.com wrote: Hi, I'm interested in working with you on the IME support issue, especially on Linux. I just read attached document, and following are some of my questions and information: 1. How about the support of on-the-spot mode (preedit/composition) and surrounding text? Which require more events than key press/release and commit. As I know that some of input methods (at least on Linux) rely on these features. 2. On Linux/Gtk, some input methods (like SCIM) will intercept key events by using gtk_key_snooper_install(), so that when input method is on, only key events that are not handled by input method will be delivered to application. Which might lead problem if application relies on those key events to work. Thank you for your comments. They are definitely helpful. To handle IME events on Linux (GTK), we notice we should not only handle more events of GTKIMContext (e.g. preedit_start, preedit_changed, preedit_end, etc.) but also decide what IPC messages we should send when an IME intercepts key events. I'm adding more diagrams that describe such CJK cases (on Windows, Mac, and Linux) to my document so that we can also figure out CJK problems and find solutions that make all platforms happy. I'm sure we should keep you informed about this issue, so it is helpful to keep giving us your comments and suggestions. Regards, Hironori Bono E-mail: hb...@chromium.org --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi Hironori, Thanks for the detailed report. I haven't nearly the same insight into the text input system as you and Avi do, but from what I've seen it may be possible for us to do at least a marginally better job using the built-in system routines and thus avoiding many sticky bugs e.g. http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts should be fixed by a CL I'm working on at the moment. The solution is to go through the native Cocoa methods to have the menu item fire the appropriate objc selector on the RHWV rather than trying to process the raw keystroke in the renderer process. I also think we can get Cocoa to translate emacs editing commands (such as cntrl-a) into the appropriate WebKit editor commands for us. Best regards, Jeremy 2009/6/5 Hironori Bono (坊野 博典) hb...@chromium.org Hi Jeremy, et al, Sorry for my slow update. This is the draft report of my investigation that implements the NSTextInput protocol on Mac, implements the GtkIMContext on Linux and observes keyboard events on Windows, Mac, and Linux. (Sorry, I would like to send it as a PDF file because the google site somehow compresses its figures too much to make its characters unreadable.) Since I'm still finding the best solutions for the issues, any comments and suggestions are definitely welcome. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/5/29 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. Since the events belong to a specific renderer a malicious renderer process can't access events not targeted at them. Best regards, Jeremy Stack trace of hitting cntrl-t (transpose) in a text view, Cocoa [browser-process] stuff marked in bold. #0 WebCore::executeTranspose (frame=0x70b9800) at
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi, re Type ctrl-a on hebrew keyboard: On OS X, ctrl-a usually does not select all text (that's what cmd-a does) but goes to the beginning of the current line, as in emacs. But that can be overridden ( http://www.hcs.harvard.edu/~jrus/site/cocoa-text.html ). Nico 2009/6/5 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, et al, Sorry for my slow update. This is the draft report of my investigation that implements the NSTextInput protocol on Mac, implements the GtkIMContext on Linux and observes keyboard events on Windows, Mac, and Linux. (Sorry, I would like to send it as a PDF file because the google site somehow compresses its figures too much to make its characters unreadable.) Since I'm still finding the best solutions for the issues, any comments and suggestions are definitely welcome. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/5/29 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. Since the events belong to a specific renderer a malicious renderer process can't access events not targeted at them. Best regards, Jeremy Stack trace of hitting cntrl-t (transpose) in a text view, Cocoa [browser-process] stuff marked in bold. #0 WebCore::executeTranspose (frame=0x70b9800) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:961 #1 0x03702f0f in WebCore::Editor::Command::execute (this=0xbfffe2f4, paramet...@0xbfffe2ac, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1450 #2 0x037052ad in WebCore::Editor::Command::execute (this=0xbfffe2f4, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1455 #3 0x0031d3da in -[WebHTMLView(WebNSTextInputSupport) doCommandBySelector:] (self=0x6cefe90, _cmd=0x943edca4, selector=0x94423fa4) at
[chromium-dev] Re: Fixing our keyboard handling act on OSX
For what it's worth, the current Linux key-handling code was not written too carefully -- we've kinda tweaked it until it seemed to work without analyzing what Windows does too carefully. Your document is very helpful, and we'll certainly take any recommendations you have on how to fix it since you now understand it better than I do. To answer some of your questions: the last time I understood any of this, I found the intent of the WebInputEvent structure was to be platform-neutral. So I also thought it's better to resolve IME differences in the front-end. But our code used to also send char events from the front-end, and now that I look at your diagrams it seems it no longer does that? I looked around at some of the code and I see it's all been changed. From webkit/api/WebInputEvent.h: // There are two schemes used for keyboard input. On Windows (and, // interestingly enough, on Mac Carbon) there are two events for a // keypress. One is a raw keydown, which provides the keycode only. // If the app doesn't handle that, then the system runs key translation // to create an event containing the generated character and pumps that // event. In such a scheme, those two events are translated to // RAW_KEY_DOWN and CHAR events respectively. In Cocoa and Gtk, key // events contain both the keycode and any translation into actual // text. In such a case, WebCore will eventually need to split the // events (see disambiguateKeyDownEvent and its callers) but we don't // worry about that here. We just use a different type (KEY_DOWN) to // indicate this. Since WebInputEvent is what we send via IPC, this comment indicates that we now rely on WebCore to construct the CHAR events from the native events provided by Mac/GTK. This makes me think it's better to finish the IME work in the back-end too. Since you probably cannot serialize a native IME event, though, you will probably need to do work on both the front and back side. Sorry for not having any real conclusions... 2009/6/5 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, et al, Sorry for my slow update. This is the draft report of my investigation that implements the NSTextInput protocol on Mac, implements the GtkIMContext on Linux and observes keyboard events on Windows, Mac, and Linux. (Sorry, I would like to send it as a PDF file because the google site somehow compresses its figures too much to make its characters unreadable.) Since I'm still finding the best solutions for the issues, any comments and suggestions are definitely welcome. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/5/29 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Very interesting writeup. Some thoughts: You don't touch on how Safari on the Mac handles this case. It seems to overlap with some of the work that Jeremy was doing based on his discoveries in WebKit's WebHTMLView.mm. Where does its _interceptEditingKeyEvent get called? Synthesizing CHAR events is an interesting idea, but how would that interact with the WebCore's handling of events? In particular, how would it interact with the fact that WebCore also synthesizes CHAR events in disambiguateKeyDown()? Avi 2009/6/5 Hironori Bono (坊野 博典) hb...@chromium.org Hi Jeremy, et al, Sorry for my slow update. This is the draft report of my investigation that implements the NSTextInput protocol on Mac, implements the GtkIMContext on Linux and observes keyboard events on Windows, Mac, and Linux. (Sorry, I would like to send it as a PDF file because the google site somehow compresses its figures too much to make its characters unreadable.) Since I'm still finding the best solutions for the issues, any comments and suggestions are definitely welcome. Regards, Hironori Bono E-mail: hb...@chromium.org 2009/5/29 Hironori Bono (坊野 博典) hb...@chromium.org: Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. Since the events belong to a specific renderer a malicious renderer process can't access events not targeted at them. Best regards, Jeremy Stack trace of hitting cntrl-t (transpose) in a text view, Cocoa [browser-process] stuff marked in bold. #0 WebCore::executeTranspose (frame=0x70b9800) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:961 #1 0x03702f0f in WebCore::Editor::Command::execute (this=0xbfffe2f4, paramet...@0xbfffe2ac, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1450 #2 0x037052ad in
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. Since the events belong to a specific renderer a malicious renderer process can't access events not targeted at them. Best regards, Jeremy Stack trace of hitting cntrl-t (transpose) in a text view, Cocoa [browser-process] stuff marked in bold. #0 WebCore::executeTranspose (frame=0x70b9800) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:961 #1 0x03702f0f in WebCore::Editor::Command::execute (this=0xbfffe2f4, paramet...@0xbfffe2ac, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1450 #2 0x037052ad in WebCore::Editor::Command::execute (this=0xbfffe2f4, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1455 #3 0x0031d3da in -[WebHTMLView(WebNSTextInputSupport) doCommandBySelector:] (self=0x6cefe90, _cmd=0x943edca4, selector=0x94423fa4) at /Users/Shared/playmobil/work/git.WebKit/WebKit/mac/WebView/WebHTMLView.mm:5337 #4 0x0031c238 in -[WebHTMLView(WebInternal) _interceptEditingKeyEvent:shouldSaveCommand:] (self=0x6cefe90, _cmd=0x3a405c, event=0x24178380, shouldSave=0 '\000') at /Users/Shared/playmobil/work/git.WebKit/WebKit/mac/WebView/WebHTMLView.mm:4966 #5 0x002eb19b in WebEditorClient::handleKeyboardEvent (this=0x6c31aa0, event=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebKit/mac/WebCoreSupport/WebEditorClient.mm:440 #6 0x036f8875 in WebCore::Editor::handleKeyboardEvent (this=0x70b9cd4, event=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/Editor.cpp:105 #7 0x037156fb in WebCore::EventHandler::defaultKeyboardEventHandler (this=0x70b9d00, event=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/page/EventHandler.cpp:1907 #8 0x03a7876e in WebCore::Node::defaultEventHandler (this=0x1c23f430, event=0x24178380) at
[chromium-dev] Re: Fixing our keyboard handling act on OSX
(Wondering if I should take the lack of response here as a resounding vote of confidence regarding the flawless design proposal?) Thanks for the reply Hironori, Since this proposal contains facilities for pingponging event handling between the browser and renderer even if it doesn't solve the IME issues directly it should provide common infrastructure to build on. I look forward to seeing your writeup, but I think these are orthogonal issues. Please correct me if I'm wrong [which is a definite possibility]. Best regards, Jeremy 2009/5/28 Hironori Bono (坊野 博典) hb...@chromium.org Hi Jeremy, I have been investigating keyboard events on Windows, Linux, and Mac to find solutions for these issues. Shall we have a discussion about these issues when I finish writing my report (maybe sometime next week)? Regards, Hironori Bono E-mail: hb...@chromium.org On Thu, May 28, 2009 at 7:14 AM, Jeremy Moskovich jer...@chromium.org wrote: Hi All, We currently fudge our keyboard handling on OSX, we interpret command key shortcuts ourselves and thus miss out on quite a few Cocoa text handling niceities. We also don't support IMEs. Relevant bugs: http://crbug.com/10862 - OS X: Can't use command-key shortcuts with foreign keyboard layouts http://crbug.com/12698 - standard macintosh text editing keyboard shortcuts are missing http://crbug.com/11981 - Deadkeys do not work http://crbug.com/11952 - IME support (Chinese input method doesnt work) We also don't handle activation/deactivation of edit menu items correctly (select all is always disabled). I've done a little digging in WebKit's keyboard handling and what follows is a proposal on how we can more closely match the OSX keyboard handling code and [hopefully] do things the right way. I don't purport to understand this all and there may be simpler ways to achieve this so feel free to comment. Standard commands (Copy/Paste/Select-all): WebKit handles these through the regular obj-c selectors in WebHTMLView, see the WEBCORE_COMMAND macro. We could do the same thing and add these selectors to BrowserWindowController which would then send an IPC message over to the renderer process to execute the appropriate command. The tricky bit is updating the menus, the model in Cocoa is for the OS to call you for each menu item before displaying a menu. We can't block the browser process on the renderer process so the browser process would always need to know if there is an active selection. WebCore appears to already have a mechanism to do this via notifyAccessibilityForSelectionChange() - we could use the same or a similar mechanism to send an IPC message back to the browser process each time the selection changes. Emacs keyboard commands and IMEs: The IME part of the title may be nonsense, but looking at the code it appears the code path and issues are the same. I've attached the stack trace for hitting cntrl-t to the end of this email. The tricky bit here is that WebCore is first given a shot at handling the event and then passes it back to WebHTMLView to take another look at the event which is where it's actually parsed. I assume this is to give JS code a chance to listen on these events. Since we can't serialize an NSEvent, we can't replicate this code solely in the renderer. A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. Since the events belong to a specific renderer a malicious renderer process can't access events not targeted at them. Best regards, Jeremy Stack trace of hitting cntrl-t (transpose) in a text view, Cocoa [browser-process] stuff marked in bold. #0 WebCore::executeTranspose (frame=0x70b9800) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:961 #1 0x03702f0f in WebCore::Editor::Command::execute (this=0xbfffe2f4, paramet...@0xbfffe2ac, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1450 #2 0x037052ad in WebCore::Editor::Command::execute (this=0xbfffe2f4, triggeringEvent=0x24178380) at /Users/Shared/playmobil/work/git.WebKit/WebCore/editing/EditorCommand.cpp:1455 #3 0x0031d3da in -[WebHTMLView(WebNSTextInputSupport) doCommandBySelector:] (self=0x6cefe90, _cmd=0x943edca4, selector=0x94423fa4) at /Users/Shared/playmobil/work/git.WebKit/WebKit/mac/WebView/WebHTMLView.mm:5337 #4 0x0031c238 in -[WebHTMLView(WebInternal) _interceptEditingKeyEvent:shouldSaveCommand:] (self=0x6cefe90, _cmd=0x3a405c, event=0x24178380, shouldSave=0 '\000') at
[chromium-dev] Re: Fixing our keyboard handling act on OSX
On Wed, May 27, 2009 at 3:14 PM, Jeremy Moskovich jer...@chromium.org wrote: A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. BTW, we also want this on the Linux side: http://code.google.com/p/chromium/issues/detail?id=11480 From browsing http://library.gnome.org/devel/gtk/stable/gtk-Bindings.html it looks like we might be able to root around in the bindings data structure to map a key event into a command name + args that could be serialized over IPC. In brief, if/when you get around to implementing such a thing, please involve Linux since we'll probably want to share some code. :) --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Thanks for the comments Avi, The reason this might relate to IME is that the WebKit code seems to handle IMEs along the same codepath. Once Hironori is done with his investigation he should be able to chime in and let us know if this is the case. Unless more important things come up I intend to start working on this pretty soon. Evan: I'll cc you on all relevant code reviews, that way we'll hopefully be able to create some good cross-platform plumbing for this stuff. Best regards, Jeremy On Thu, May 28, 2009 at 2:58 PM, Evan Martin e...@chromium.org wrote: On Wed, May 27, 2009 at 3:14 PM, Jeremy Moskovich jer...@chromium.org wrote: A possible solution to this would be to store a queue of the last N NSEvents per renderer matched with an ID. the event would then be serialized and sent to the renderer which could then send it's own IPC message back to the browser process to get Cocoa to handle the message, we could pick the NSEvent out of the queue by ID and send back the relevant edit command to the renderer. BTW, we also want this on the Linux side: http://code.google.com/p/chromium/issues/detail?id=11480 From browsing http://library.gnome.org/devel/gtk/stable/gtk-Bindings.html it looks like we might be able to root around in the bindings data structure to map a key event into a command name + args that could be serialized over IPC. In brief, if/when you get around to implementing such a thing, please involve Linux since we'll probably want to share some code. :) --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Fixing our keyboard handling act on OSX
Avi, On Fri, May 29, 2009 at 5:45 AM, Avi Drissman a...@chromium.org wrote: We already do much of this. Take a look at TabContentsViewMac::HandleKeyboardEvent, where ids referring to keyboard events not handled by WebKit are returned to the browser process. TabContentsViewMac then retrieves the original event and passes it up to Cocoa for handling. That might be an opportune time to Do The Right Thing. How do IMEs belong here? In Mac (Cocoa) (*1), IME events are sent through the NSTextInput interface. So, we need to implement the interface to support IMEs. I have added a mock implementation to monitor IME events on Mac. (This implementation is still in an infant phase, though.) (*1) I assume we don't have to support Carbon, right? In Linux, IME events are sent though GtkIMContext signals as well as some keyboard events from non-US keyboards, e.g. dead keys, etc. As I wrote in my previous reply, I have been monitoring all keyboard-related events including IME ones on Windows, Linux, and Mac to compare them side-by-side and find possible solutions. I hope I can send a report about my investigation sometime next week. Regards, Hironori Bono E-mail: hb...@chromium.org --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
