Re: Complex text input in Plasma
Hello list, I was not following this list but the discussion was interesting, so I am adding my opinion/personal mumblings/etc. to this topic. For entering Korean texts, the expectations and requirements of IMs are different from Chinese/Japanese. As most multilingual IM cores are designed by C/J community, the difference is sometimes overlooked. Unlike C/J where all composing is explicit, Korean users are expecting implicit commits when the textbox loses focus. Changing this behavior is not acceptable as this is what we expected from the beginning of PC. Some IMs did not handled this behavior correctly in past, and this was not clearly solved for several years. [1] [1] https://code.google.com/archive/p/ibus/issues/1726 and https:// bugreports.qt.io/browse/QTBUG-40541 Some Korean users (including me) are thinking current IMs in Linux desktops (fcitx, ibus, etc.) are somewhat overbloated, since entering Korean texts does not really require the typical features used by C/J ((personalized) dictionaries are not typically used, numbers and punctuations are always half- width). Even though I am using fcitx right now, this change was somewhat "forced" as Qt 5 dropped XIM support, and I had to abandon the good-old Nabi who supported just XIM and handled only Korean. Martin Gräßlin: > I dislike all three options. The base keyboard layout handling must be > in KWin. KWin has to control the IM stack, not the other way around. > > I'm not giving away control over such fundamental aspects of the stack > any more. I want that these things function correctly and I'm not > trusting 3rd party software to do the right things. +1 from my side to be consistent. I enabled Korean, Russian and German keyboard layout via systemsettings, and expecting IM is only enabled when the layout is set to Korean. I had to configure those in fcitx also to make it consistent, and sometimes keyboard layout got scrambled upon changing window and have to manually reset the layout as I want. And I also had to configure fcitx to respect my xkb options (and do not reset it upon changing IM) too. Martin Gräßlin: > That would be an option if we want to support multiple IM > implementations. > > But if we want to keep it small and simple I would go to directly link > the fcitx library and do the deep calls into fcitx directly from KWin > core. I am opposing integrating specific IM into KWin core as we don't know how IMs and its APIs will change in future. There were quite a lot of IMs throughout the history (scim, ibus, fcitx, uim, ...) which changed from time to time. But if a stability in functionality and API is guaranteed by integrating some IM into KWin core I would not object it. I am personally reluctant to change IMs, as it might introduce a new bug in handling texts, and Korean plugin for all those IMs are almost the same in functionality. Shinjo
Re: Complex text input in Plasma
On 04/07/17 22:46, Martin Gräßlin-san wrote: Am 2017-04-07 07:56, schrieb Takao Fujiwara: Due to that: no chance for IM controlling part of our stack. We control the IM. Probably I think this way would not work with IBus. Each IBus IME are called by IBus dbus method. You hardly ask each IME maintainer to change the protocol. IBus daemon would be the controller of IBUs IMEs. I think I need to describe in a better way how I envision the future architecture. I want to have the IM daemon merged into the wayland compositor. So KWin would talk directly with the IM through library calls and the composed text would be sent from KWin to the application through the Wayland text-input protocol. Probably I'd think KWin does not have to merge IM daemon and I hope we can make DBus proceses more secure. Also I think virtual keyboards does not need to implement IM and they can enable IM with the shortcuts outside IM implementations. Probably you have to support copy & paste between applications and support protocols between processes. I want to eliminate the IPC between applications and IM daemon. If we are going to touch this code we can strive for the best solution and not keep stuck on the approach used on X11. And yes that might need adjusting the IMEs. But they need to be adjusted anyway. We wouldn't have this discussion if it would just work on Wayland. I guess just implementing Plasma Wayland with the current IM modules also will causes many regressions as GNOME did. Thanks, Fujiwara Cheers Martin
Re: Complex text input in Plasma
On 04/09/17 01:46, Martin Gräßlin-san wrote: Am 2017-04-08 17:26, schrieb Weng Xuetian: You're wrong about the QT_IM_MODULE stuff. To make application to use the wayland protocol to type (text-input), the implementation must be done with QT_IM_MODULE=wayland. I don't mind if it is set to certain application but set it in general won't work. Also to have real virtual keyboard , you need to let the input method daemon to provides a virtual keyboard implementation. No you are wrong about that one :-) It might be that it used to be like that, but Wayland is the default if no QT_IM_MODULE is specified. See https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegration.cpp#n142 And also, merging more and more daemon into kwin is not always good even from security point of view. The problem is, once it got merged, the whole memory space is being exposed. Which means, if there's a single piece of code is vulnerable, it will affect the whole compositor. We are not perfect people, and that's why put more code together will make it more vulnerable to attacker. If you consider that, your prevention of ptrace on kwin becomes nothing and so does your effort to make kwin not loading some random plugin (prevent ld_preload and qt_plugins_path?). The security of the system breaks with the weakest link. Whether the IM daemon is insecure by running standalone or inside KWin isn't a difference. I think the weakest link still can make more secure in the protocol. I'd suggest to make Plasma Wayland with the current IM modules in the first stage since I guess there will be many regressions in Plasma Wayland. Currently I'm also involved a discussion how to secure the DBus in the GNOME side and I also think it could be utilized for Plasma Wayland. Otherwise I think it might be difficult to support Plasma in some distributions. Wayland staffs has contributed in the IBus and GNOME connection for a long time ago. https://github.com/ibus/ibus/tree/master/client/wayland But it's not used in GNOME yet.
Re: Complex text input in Plasma
On Sunday, 9 April 2017 00:38:06 PDT，Martin Gräßlin wrote： > Am 2017-04-09 03:49, schrieb Weng Xuetian: > >> I don't want to base any decisions on 5 year old numbers. 30 msec > >> would > >> be bad, but if it is with modern hardware just 10 it wouldn't be that > >> bad. > > > > 30ms is almost the maximum time that I found it would be accepted by > > the user. > > It's too much already. 30 msec means 2 frames, that is something users > already start to see. > > > But it also means that if it is ok, we could put more computation into > > it to > > achieve better result. Also, I do feel that synchronously wait for > > input > > method result would be bad. In our customized im module era, we are > > also > > moving to synchronous wait to key event processing result to > > asynchronous. As > > you said their exists same problem, but overall it improves the > > responsivieness when something does go wrong (when it's not a bug it > > could be > > memory or disk io). > > > >> > Also the "input method always on" idea is not saying that every single > >> > key > >> > will be send to input method, it just mean whenever user need type > >> > text, it > >> > will be forwarded to input method. > >> > > >> > I don't know whether I could persuade you, but please look around the > >> > world, > >> > iOS, android, sailfish (also use wayland), mac OS , (windows used to > >> > run im > >> > directly inside application.. but now they also have a service for that > >> > ), no > >> > one would run a im daemon inside the compositor. Are they all ignore > >> > the ipc > >> > overhead? > >> > >> I'm using sailfish and Android and both keyboards suck bad times. > >> Sailfish is really bad, well it's maliit so no surprise there. With > >> both > >> systems I see problems in UI positioning because it's not in the > >> compositor. Both don't feel perfect especially not in the "every frame > >> perfect sense" we are aiming for on Wayland. > > > > But afterall, running fcitx as a library is possible since it is how it > > is > > implemented. Technically there's no such difficulty prevent us to do > > so. And I > > have been using fcitx as a library for testing purpose. But I guess > > the story > > is different for ibus, ibus is multi-process and its engine always > > running in > > different process, and it will always need to pass some ipc to its own > > engine. > > > > On the bright side, I could bypass the ugly zwp_input_method_v1 and > > directly > > have zwp_text_input_v1 (or even unimplemented v2), I'd love to see > > that. > > hehe, I think we can agree on not liking that interface ;-) > > > While there's still some hidden problem, the input method upgrade won't > > be > > easy since it's implmented via shared library (dlclose is dangerous > > that I > > always disable it via DL_NODELETE), I don't know if it would be > > bothersome to > > user or not, but luckily it's not something that will receive updates > > everyday. > > Right it would mean session restart to upgrade. But even with dedicated > IM daemon it would mean session restart. As for this, not necessarily. Nowadays, it is totally fine to restart im daemon itself without breaking the desktop (Except the crappy XIM Xlib implementation which is synchronous everywhere and will choke application on the certain function). Weston also has some watchdog stuff to monitor whether the im daemon is dead and restart it. > > If we go for not shared library approach KWin would have to start the IM > daemon and pass it a socket. Only the process started by KWin would be > allowed to bind the zwp_input_method_v1 interface - or better said: only > that process would be granted access to it. Given that upgrading would > not be trivial either. > > Anyway going through the interface would be possible as well. It would > require some special handling in KWin, but we do have comparable special > handling for maliit for the Plasma phone. > > What we would need is implement support for zwp_input_method_v1 in > KWayland, then add support for it in KWin. > > Cheers > Martin
Re: Complex text input in Plasma
Am 2017-04-09 10:19, schrieb Eike Hein: Martin, how would you see this done? Some sort of abstract interface in kwin and a plugin-based implementation that makes the actual calls into the fcitx lib? That would be an option if we want to support multiple IM implementations. But if we want to keep it small and simple I would go to directly link the fcitx library and do the deep calls into fcitx directly from KWin core. This has the charm that we can move relatively fast and enforce that distros have to ship with it. It would become a mandatory dependency and there would not be any weird "is the plugin installed" logic required. Also changing to plugin API could be done in a later step. We just need to keep the KWin internal interface clean ;-) But as we have most of the required code anyway for the virtual keyboard I'm not worried about that. Cheers Martin
Re: Complex text input in Plasma
Personally I do see the appeal of making kwin the daemon - simplicity and definitely delivers on the "make IM not be an afterthought" goal. If it's in the kwin core it can't be not installed. It'd also make the System Settings side easier if we can code against a native interface we control - right now an ibus/fcitx KCM bundled with Plasma Desktop could break if those break API. If what we're configuring is kwin, it reduces the API surface area toward our deps. OTOH I think we need something like fcitx (or ibus) between us and the IME instead of having a native IME plugin API in kwin: There's only a limited number of people developing IMEs across all free systems, and it's important to maximize their utility by allowing them to get their IMEs working in as many systems as possible. Martin, how would you see this done? Some sort of abstract interface in kwin and a plugin-based implementation that makes the actual calls into the fcitx lib? > Cheers > Martin Cheers, Eike
Re: Complex text input in Plasma
Am 2017-04-09 03:49, schrieb Weng Xuetian: I don't want to base any decisions on 5 year old numbers. 30 msec would be bad, but if it is with modern hardware just 10 it wouldn't be that bad. 30ms is almost the maximum time that I found it would be accepted by the user. It's too much already. 30 msec means 2 frames, that is something users already start to see. But it also means that if it is ok, we could put more computation into it to achieve better result. Also, I do feel that synchronously wait for input method result would be bad. In our customized im module era, we are also moving to synchronous wait to key event processing result to asynchronous. As you said their exists same problem, but overall it improves the responsivieness when something does go wrong (when it's not a bug it could be memory or disk io). > Also the "input method always on" idea is not saying that every single > key > will be send to input method, it just mean whenever user need type > text, it > will be forwarded to input method. > > I don't know whether I could persuade you, but please look around the > world, > iOS, android, sailfish (also use wayland), mac OS , (windows used to > run im > directly inside application.. but now they also have a service for that > ), no > one would run a im daemon inside the compositor. Are they all ignore > the ipc > overhead? I'm using sailfish and Android and both keyboards suck bad times. Sailfish is really bad, well it's maliit so no surprise there. With both systems I see problems in UI positioning because it's not in the compositor. Both don't feel perfect especially not in the "every frame perfect sense" we are aiming for on Wayland. But afterall, running fcitx as a library is possible since it is how it is implemented. Technically there's no such difficulty prevent us to do so. And I have been using fcitx as a library for testing purpose. But I guess the story is different for ibus, ibus is multi-process and its engine always running in different process, and it will always need to pass some ipc to its own engine. On the bright side, I could bypass the ugly zwp_input_method_v1 and directly have zwp_text_input_v1 (or even unimplemented v2), I'd love to see that. hehe, I think we can agree on not liking that interface ;-) While there's still some hidden problem, the input method upgrade won't be easy since it's implmented via shared library (dlclose is dangerous that I always disable it via DL_NODELETE), I don't know if it would be bothersome to user or not, but luckily it's not something that will receive updates everyday. Right it would mean session restart to upgrade. But even with dedicated IM daemon it would mean session restart. If we go for not shared library approach KWin would have to start the IM daemon and pass it a socket. Only the process started by KWin would be allowed to bind the zwp_input_method_v1 interface - or better said: only that process would be granted access to it. Given that upgrading would not be trivial either. Anyway going through the interface would be possible as well. It would require some special handling in KWin, but we do have comparable special handling for maliit for the Plasma phone. What we would need is implement support for zwp_input_method_v1 in KWayland, then add support for it in KWin. Cheers Martin
Re: Complex text input in Plasma
On Saturday, 8 April 2017 12:43:54 PDT，Martin Gräßlin wrote： > Am 2017-04-08 21:21, schrieb Weng Xuetian: > > On Saturday, 8 April 2017 09:46:17 PDT，Martin Gräßlin wrote： > > > >> Am 2017-04-08 17:26, schrieb Weng Xuetian: > >> > You're wrong about the QT_IM_MODULE stuff. To make application to use > >> > the > >> > wayland protocol to type (text-input), the implementation must be done > >> > with > >> > QT_IM_MODULE=wayland. I don't mind if it is set to certain application > >> > but set > >> > it in general won't work. Also to have real virtual keyboard , you need > >> > to let > >> > the input method daemon to provides a virtual keyboard implementation. > >> > >> No you are wrong about that one :-) It might be that it used to be > >> like > >> that, but Wayland is the default if no QT_IM_MODULE is specified. See > >> https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegrat > >> ion .cpp#n142 > > > > What I want to say is, if you set QT_IM_MODULE to qtvirtualkeyboard, > > that app > > can't talk to im daemon via the wayland protocol. The automatic > > selection > > procedure is not really a problem there. > > And that's not the case. Our virtual keyboard is using the Wayland text > input protocol and does not require setting the QT_IM_MODULE at all. > > Please believe a little bit in what I'm writing. I coded this stuff. > > > Also, running it inside kwin could still be problematic. It'll also > > need to > > display UI. From what I heard about, to run a qml window inside kwin > > could > > even be a problem. > > I don't know what you have heard but all our UI in KWin is written in > Qml nowadays. > > > The races you pointed about is not really a problem. Weston has already > > done > > it well. Unless you want to reinvent a new protocol, using the current > > wayland > > protocol essentially does what I said. And if such a thing would > > interfere > > with kwin's rendering procedure, then probably kwin should refactor > > about > > that. > > Just because Weston doesn't think of this problem doesn't mean it > doesn't exist. Weston is not perfect and I found several bugs in the way > they implemented the protocols they developed and Weston is the > reference for. > > > And let me give you some number here, a modern input method for Chinese > > could > > takes about 30ms to handle one single key (it is THAT complex and we > > are all > > limited by the complexity, though I measured it about 5 years ago). You > > really > > want to have kwin block on that and wait for such a long time? And just > > because it is such a slow thing, the IPC latency is not even > > comparable. If > > you want good prediction or handwriting, it will be slow. There's no im > > use > > neural network on linux right now, but iOS's virtual keyboard is > > already doing > > that. Which might be the future, but also mean the im processing time > > might > > not slow down. > > I don't want to base any decisions on 5 year old numbers. 30 msec would > be bad, but if it is with modern hardware just 10 it wouldn't be that > bad. 30ms is almost the maximum time that I found it would be accepted by the user. But it also means that if it is ok, we could put more computation into it to achieve better result. Also, I do feel that synchronously wait for input method result would be bad. In our customized im module era, we are also moving to synchronous wait to key event processing result to asynchronous. As you said their exists same problem, but overall it improves the responsivieness when something does go wrong (when it's not a bug it could be memory or disk io). > > > Also the "input method always on" idea is not saying that every single > > key > > will be send to input method, it just mean whenever user need type > > text, it > > will be forwarded to input method. > > > > I don't know whether I could persuade you, but please look around the > > world, > > iOS, android, sailfish (also use wayland), mac OS , (windows used to > > run im > > directly inside application.. but now they also have a service for that > > ), no > > one would run a im daemon inside the compositor. Are they all ignore > > the ipc > > overhead? > > I'm using sailfish and Android and both keyboards suck bad times. > Sailfish is really bad, well it's maliit so no surprise there. With both > systems I see problems in UI positioning because it's not in the > compositor. Both don't feel perfect especially not in the "every frame > perfect sense" we are aiming for on Wayland. > But afterall, running fcitx as a library is possible since it is how it is implemented. Technically there's no such difficulty prevent us to do so. And I have been using fcitx as a library for testing purpose. But I guess the story is different for ibus, ibus is multi-process and its engine always running in different process, and it will always need to pass some ipc to its own engine. On the bright side, I could bypass the ugly zwp_input_method_v1
Re: Complex text input in Plasma
Am 2017-04-08 21:21, schrieb Weng Xuetian: On Saturday, 8 April 2017 09:46:17 PDT，Martin Gräßlin wrote： Am 2017-04-08 17:26, schrieb Weng Xuetian: > You're wrong about the QT_IM_MODULE stuff. To make application to use > the > wayland protocol to type (text-input), the implementation must be done > with > QT_IM_MODULE=wayland. I don't mind if it is set to certain application > but set > it in general won't work. Also to have real virtual keyboard , you need > to let > the input method daemon to provides a virtual keyboard implementation. No you are wrong about that one :-) It might be that it used to be like that, but Wayland is the default if no QT_IM_MODULE is specified. See https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegration .cpp#n142 What I want to say is, if you set QT_IM_MODULE to qtvirtualkeyboard, that app can't talk to im daemon via the wayland protocol. The automatic selection procedure is not really a problem there. And that's not the case. Our virtual keyboard is using the Wayland text input protocol and does not require setting the QT_IM_MODULE at all. Please believe a little bit in what I'm writing. I coded this stuff. Also, running it inside kwin could still be problematic. It'll also need to display UI. From what I heard about, to run a qml window inside kwin could even be a problem. I don't know what you have heard but all our UI in KWin is written in Qml nowadays. The races you pointed about is not really a problem. Weston has already done it well. Unless you want to reinvent a new protocol, using the current wayland protocol essentially does what I said. And if such a thing would interfere with kwin's rendering procedure, then probably kwin should refactor about that. Just because Weston doesn't think of this problem doesn't mean it doesn't exist. Weston is not perfect and I found several bugs in the way they implemented the protocols they developed and Weston is the reference for. And let me give you some number here, a modern input method for Chinese could takes about 30ms to handle one single key (it is THAT complex and we are all limited by the complexity, though I measured it about 5 years ago). You really want to have kwin block on that and wait for such a long time? And just because it is such a slow thing, the IPC latency is not even comparable. If you want good prediction or handwriting, it will be slow. There's no im use neural network on linux right now, but iOS's virtual keyboard is already doing that. Which might be the future, but also mean the im processing time might not slow down. I don't want to base any decisions on 5 year old numbers. 30 msec would be bad, but if it is with modern hardware just 10 it wouldn't be that bad. Also the "input method always on" idea is not saying that every single key will be send to input method, it just mean whenever user need type text, it will be forwarded to input method. I don't know whether I could persuade you, but please look around the world, iOS, android, sailfish (also use wayland), mac OS , (windows used to run im directly inside application.. but now they also have a service for that ), no one would run a im daemon inside the compositor. Are they all ignore the ipc overhead? I'm using sailfish and Android and both keyboards suck bad times. Sailfish is really bad, well it's maliit so no surprise there. With both systems I see problems in UI positioning because it's not in the compositor. Both don't feel perfect especially not in the "every frame perfect sense" we are aiming for on Wayland. Cheers Martin
Re: Complex text input in Plasma
On Saturday, 8 April 2017 09:46:17 PDT，Martin Gräßlin wrote： > Am 2017-04-08 17:26, schrieb Weng Xuetian: > > You're wrong about the QT_IM_MODULE stuff. To make application to use > > the > > wayland protocol to type (text-input), the implementation must be done > > with > > QT_IM_MODULE=wayland. I don't mind if it is set to certain application > > but set > > it in general won't work. Also to have real virtual keyboard , you need > > to let > > the input method daemon to provides a virtual keyboard implementation. > > No you are wrong about that one :-) It might be that it used to be like > that, but Wayland is the default if no QT_IM_MODULE is specified. See > https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegration > .cpp#n142 What I want to say is, if you set QT_IM_MODULE to qtvirtualkeyboard, that app can't talk to im daemon via the wayland protocol. The automatic selection procedure is not really a problem there. > > And also, merging more and more daemon into kwin is not always good > > even from > > security point of view. The problem is, once it got merged, the whole > > memory > > space is being exposed. Which means, if there's a single piece of code > > is > > vulnerable, it will affect the whole compositor. We are not perfect > > people, and > > that's why put more code together will make it more vulnerable to > > attacker. If > > you consider that, your prevention of ptrace on kwin becomes nothing > > and so > > does your effort to make kwin not loading some random plugin (prevent > > ld_preload and qt_plugins_path?). > > The security of the system breaks with the weakest link. Whether the IM > daemon is insecure by running standalone or inside KWin isn't a > difference. > > > So, my proposal to this will be: > > im daemon <-> kwin <-> application, and the communication is done with > > some > > wayland protocol. > > > > and imho that would be best option to all people. Here's the reason > > > > Pros > > - less code to run within kwin process > > - less change needed to existing im daemon (we just need an extra > > frontend) > > and kwin > > - no hard dependency > > > > Cons: > > more ipc latency, but this only happens if user need to type. > > The IPC latency is a huge problem here. If we want to have a good > experience KWin needs to decide whether to send a key event to the IM > daemon or through wl_keyboard interface to the application. This means > possibly a roundtrip in the event handling code. This code is currently > the most crucial part of KWin. It opens a can of worms if we go IPC > there and given your suggestion it needs IPC. > > And even if we handle it without roundtrip we run into timing issues. > Consider: > * KWin gets key event > * KWin decides it goes to IM for window foo > * IM does something with it > * window foo closes > * IM sends the composed text for window foo to KWin > > Now what? In case we would eliminate the IPC by making KWin link the IM > module it becomes: > > * KWin gets key event > * KWin decides it goes to IM for window foo > * IM returns the composed text for window foo > * KWin forwards the composed text to window foo > * window foo closes > > So from KWin perspective everything becomes cleaner if there is no IPC > involved towards the IM daemon. In that case KWin needs to adjust way > less as we don't need a dedicated protocol. > > I understand your concerns with linking stuff and loading plugins but I > don't share them. Yes I also see this as a problem but I think it's less > of a problem than IPC in this crucial area. Then why IPC exists in the first place? The whole wayland stuff IS IPC. Why there ARE processes?.. Also, running it inside kwin could still be problematic. It'll also need to display UI. From what I heard about, to run a qml window inside kwin could even be a problem. The races you pointed about is not really a problem. Weston has already done it well. Unless you want to reinvent a new protocol, using the current wayland protocol essentially does what I said. And if such a thing would interfere with kwin's rendering procedure, then probably kwin should refactor about that. And let me give you some number here, a modern input method for Chinese could takes about 30ms to handle one single key (it is THAT complex and we are all limited by the complexity, though I measured it about 5 years ago). You really want to have kwin block on that and wait for such a long time? And just because it is such a slow thing, the IPC latency is not even comparable. If you want good prediction or handwriting, it will be slow. There's no im use neural network on linux right now, but iOS's virtual keyboard is already doing that. Which might be the future, but also mean the im processing time might not slow down. Also the "input method always on" idea is not saying that every single key will be send to input method, it just mean whenever user need type text, it will be forwarded to
Re: Complex text input in Plasma
Am 2017-04-08 17:26, schrieb Weng Xuetian: You're wrong about the QT_IM_MODULE stuff. To make application to use the wayland protocol to type (text-input), the implementation must be done with QT_IM_MODULE=wayland. I don't mind if it is set to certain application but set it in general won't work. Also to have real virtual keyboard , you need to let the input method daemon to provides a virtual keyboard implementation. No you are wrong about that one :-) It might be that it used to be like that, but Wayland is the default if no QT_IM_MODULE is specified. See https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegration.cpp#n142 And also, merging more and more daemon into kwin is not always good even from security point of view. The problem is, once it got merged, the whole memory space is being exposed. Which means, if there's a single piece of code is vulnerable, it will affect the whole compositor. We are not perfect people, and that's why put more code together will make it more vulnerable to attacker. If you consider that, your prevention of ptrace on kwin becomes nothing and so does your effort to make kwin not loading some random plugin (prevent ld_preload and qt_plugins_path?). The security of the system breaks with the weakest link. Whether the IM daemon is insecure by running standalone or inside KWin isn't a difference. So, my proposal to this will be: im daemon <-> kwin <-> application, and the communication is done with some wayland protocol. and imho that would be best option to all people. Here's the reason Pros - less code to run within kwin process - less change needed to existing im daemon (we just need an extra frontend) and kwin - no hard dependency Cons: more ipc latency, but this only happens if user need to type. The IPC latency is a huge problem here. If we want to have a good experience KWin needs to decide whether to send a key event to the IM daemon or through wl_keyboard interface to the application. This means possibly a roundtrip in the event handling code. This code is currently the most crucial part of KWin. It opens a can of worms if we go IPC there and given your suggestion it needs IPC. And even if we handle it without roundtrip we run into timing issues. Consider: * KWin gets key event * KWin decides it goes to IM for window foo * IM does something with it * window foo closes * IM sends the composed text for window foo to KWin Now what? In case we would eliminate the IPC by making KWin link the IM module it becomes: * KWin gets key event * KWin decides it goes to IM for window foo * IM returns the composed text for window foo * KWin forwards the composed text to window foo * window foo closes So from KWin perspective everything becomes cleaner if there is no IPC involved towards the IM daemon. In that case KWin needs to adjust way less as we don't need a dedicated protocol. I understand your concerns with linking stuff and loading plugins but I don't share them. Yes I also see this as a problem but I think it's less of a problem than IPC in this crucial area. Cheers Martin
Re: Complex text input in Plasma
On Friday, 7 April 2017 23:18:37 PDT，Martin Gräßlin wrote： > Am 2017-04-08 01:04, schrieb Weng Xuetian: > > On Friday, 7 April 2017 06:46:03 PDT，Martin Gräßlin wrote： > > > >> Am 2017-04-07 07:56, schrieb Takao Fujiwara: > >> >> Due to that: no chance for IM controlling part of our stack. We > >> >> control the IM. > >> > > >> > Probably I think this way would not work with IBus. > >> > Each IBus IME are called by IBus dbus method. You hardly ask each IME > >> > maintainer to change the protocol. > >> > IBus daemon would be the controller of IBUs IMEs. > >> > >> I think I need to describe in a better way how I envision the future > >> architecture. > >> > >> I want to have the IM daemon merged into the wayland compositor. So > >> KWin > >> would talk directly with the IM through library calls and the composed > >> text would be sent from KWin to the application through the Wayland > >> text-input protocol. > >> > >> I want to eliminate the IPC between applications and IM daemon. > >> > >> If we are going to touch this code we can strive for the best solution > >> and not keep stuck on the approach used on X11. And yes that might > >> need > >> adjusting the IMEs. But they need to be adjusted anyway. We wouldn't > >> have this discussion if it would just work on Wayland. > >> > >> Cheers > >> Martin > > > > I'm sorry to point out this won't really works. The Xwayland client > > will have > > to use XIM anyway. > > Let's please completely ignore the legacy world in the planning for the > future. We setup the perfect world and then look how we can make > Xwayland work. Even ignoring X11 might be an option as Wayland will now > get a massive push thanks to Mir finally being dead. > > > From security point of view, I'm ok if the daemon doesn't directly > > connect to > > application. But I don't want kwin to run the daemon within kwin's > > process. It > > would impose too much complication and I will not be comfortable about > > that. > > I don't mind the IM daemon being part of KWin. > > > People will need to load certain concrete input method as a plugin, I > > don't > > think kwin itself want to include yet another plugin complex system to > > be > > running inside kwin. > > I don't mind that either. > > > Just think it something similar, will kwin merge plasma? I don't think > > you > > really want to that. > > Well we merged: > * kglobalaccel > * screenlocker > * keyboard kded > * virtual keyboard > * multi screen functionality > * Everything that X-Server does > * ... > > No I wouldn't mind merging plasma if it would make sense and improve the > overall experience. That's what I care for: offering the best possible > and secure experience. If that requires merging yet another daemon I > don't mind. There are no sacred cows to me. We are changing the complete > architecture of the display management, so I don't have a problem with > rethinking everything. > > > So the best solution IMHO, is to let kwin to expose certain feature > > that will > > only be allowed to be accessed the im daemon that kwin starts, which > > means > > user any not arbitrary launch a daemon and do whatever they want. And > > that IS > > what current wayland's im protocol is doing. > > > > Also, talking about the layout, I don't think you really want to write > > the > > layout control code. I have never seen a developer who only use > > keyboard > > layout stuff knows what kind of feature that input method could > > achieve. > > Please note that only the Wayland compositor is able to set the keyboard > layout. That is a technical restriction of the Wayland protocol. So yes > KWin must (!) manage the keyboard layout no matter what IM devs think > about it. It just must be in KWin. > > > Compositor should really give this kind of freedom to input method > > daemon. It > > doesn't expose extra security issue if you think about it. kwin would > > just > > works like a router that talks with input method via wayland protocol. > > And > > kwin will not talk with some random input method daemon that it > > unprivileged. > > I don't mind giving IM some freedom, but at the center of the > interaction there is KWin. > > So for me the requirements are: > * no key events are sent via DBus > * applications do not talk to IM daemon > * QT_IM_MODULE must not be used as that breaks virtual keyboard > handling > * No IPC in the key event handling code of KWin > > I don't mind whether the IM daemon is a separate process only KWin > interacts with or a library KWin links. I think a library would be > better as that removes the need for IPC and allows to directly involve > the IM from keyboard handling code. Please compare in that point to > kglobalaccel. > > On X11 kglobalaccel only listens to the key events which would trigger > the shortcut and it's restricted by things like keyboard grab. On > Wayland KWin can send any key event through kglobalaccel as it's linked > directly. This allows us to make
Re: Complex text input in Plasma
Am 2017-04-08 01:04, schrieb Weng Xuetian: On Friday, 7 April 2017 06:46:03 PDT，Martin Gräßlin wrote： Am 2017-04-07 07:56, schrieb Takao Fujiwara: >> Due to that: no chance for IM controlling part of our stack. We >> control the IM. > > Probably I think this way would not work with IBus. > Each IBus IME are called by IBus dbus method. You hardly ask each IME > maintainer to change the protocol. > IBus daemon would be the controller of IBUs IMEs. I think I need to describe in a better way how I envision the future architecture. I want to have the IM daemon merged into the wayland compositor. So KWin would talk directly with the IM through library calls and the composed text would be sent from KWin to the application through the Wayland text-input protocol. I want to eliminate the IPC between applications and IM daemon. If we are going to touch this code we can strive for the best solution and not keep stuck on the approach used on X11. And yes that might need adjusting the IMEs. But they need to be adjusted anyway. We wouldn't have this discussion if it would just work on Wayland. Cheers Martin I'm sorry to point out this won't really works. The Xwayland client will have to use XIM anyway. Let's please completely ignore the legacy world in the planning for the future. We setup the perfect world and then look how we can make Xwayland work. Even ignoring X11 might be an option as Wayland will now get a massive push thanks to Mir finally being dead. From security point of view, I'm ok if the daemon doesn't directly connect to application. But I don't want kwin to run the daemon within kwin's process. It would impose too much complication and I will not be comfortable about that. I don't mind the IM daemon being part of KWin. People will need to load certain concrete input method as a plugin, I don't think kwin itself want to include yet another plugin complex system to be running inside kwin. I don't mind that either. Just think it something similar, will kwin merge plasma? I don't think you really want to that. Well we merged: * kglobalaccel * screenlocker * keyboard kded * virtual keyboard * multi screen functionality * Everything that X-Server does * ... No I wouldn't mind merging plasma if it would make sense and improve the overall experience. That's what I care for: offering the best possible and secure experience. If that requires merging yet another daemon I don't mind. There are no sacred cows to me. We are changing the complete architecture of the display management, so I don't have a problem with rethinking everything. So the best solution IMHO, is to let kwin to expose certain feature that will only be allowed to be accessed the im daemon that kwin starts, which means user any not arbitrary launch a daemon and do whatever they want. And that IS what current wayland's im protocol is doing. Also, talking about the layout, I don't think you really want to write the layout control code. I have never seen a developer who only use keyboard layout stuff knows what kind of feature that input method could achieve. Please note that only the Wayland compositor is able to set the keyboard layout. That is a technical restriction of the Wayland protocol. So yes KWin must (!) manage the keyboard layout no matter what IM devs think about it. It just must be in KWin. Compositor should really give this kind of freedom to input method daemon. It doesn't expose extra security issue if you think about it. kwin would just works like a router that talks with input method via wayland protocol. And kwin will not talk with some random input method daemon that it unprivileged. I don't mind giving IM some freedom, but at the center of the interaction there is KWin. So for me the requirements are: * no key events are sent via DBus * applications do not talk to IM daemon * QT_IM_MODULE must not be used as that breaks virtual keyboard handling * No IPC in the key event handling code of KWin I don't mind whether the IM daemon is a separate process only KWin interacts with or a library KWin links. I think a library would be better as that removes the need for IPC and allows to directly involve the IM from keyboard handling code. Please compare in that point to kglobalaccel. On X11 kglobalaccel only listens to the key events which would trigger the shortcut and it's restricted by things like keyboard grab. On Wayland KWin can send any key event through kglobalaccel as it's linked directly. This allows us to make way better decisions as we have a global view. We can decide whether right now a global shortcut makes sense or not and then not send it in. Please note that I look at this from an absolute engineer perspective. I don't care how it was in the past and only look at how it should be in the future. Especially given that the situation on X11 is bad. I don't want users to have to configure anything. If we add this to KWin
Re: Complex text input in Plasma
On Friday, 7 April 2017 06:46:03 PDT，Martin Gräßlin wrote： > Am 2017-04-07 07:56, schrieb Takao Fujiwara: > >> Due to that: no chance for IM controlling part of our stack. We > >> control the IM. > > > > Probably I think this way would not work with IBus. > > Each IBus IME are called by IBus dbus method. You hardly ask each IME > > maintainer to change the protocol. > > IBus daemon would be the controller of IBUs IMEs. > > I think I need to describe in a better way how I envision the future > architecture. > > I want to have the IM daemon merged into the wayland compositor. So KWin > would talk directly with the IM through library calls and the composed > text would be sent from KWin to the application through the Wayland > text-input protocol. > > I want to eliminate the IPC between applications and IM daemon. > > If we are going to touch this code we can strive for the best solution > and not keep stuck on the approach used on X11. And yes that might need > adjusting the IMEs. But they need to be adjusted anyway. We wouldn't > have this discussion if it would just work on Wayland. > > Cheers > Martin I'm sorry to point out this won't really works. The Xwayland client will have to use XIM anyway. From security point of view, I'm ok if the daemon doesn't directly connect to application. But I don't want kwin to run the daemon within kwin's process. It would impose too much complication and I will not be comfortable about that. People will need to load certain concrete input method as a plugin, I don't think kwin itself want to include yet another plugin complex system to be running inside kwin. Just think it something similar, will kwin merge plasma? I don't think you really want to that. So the best solution IMHO, is to let kwin to expose certain feature that will only be allowed to be accessed the im daemon that kwin starts, which means user any not arbitrary launch a daemon and do whatever they want. And that IS what current wayland's im protocol is doing. Also, talking about the layout, I don't think you really want to write the layout control code. I have never seen a developer who only use keyboard layout stuff knows what kind of feature that input method could achieve. In the real crazy world (just some random idea not really say it would be practical), I could implement a feature like: because Input method daemon knows that the text around the current cursor (which is allowed by the protocol actually), I can detect the language and switch the input method. Input method itself is already a complex system. I used to have many ideas about how to manage layout and input method together. https:// www.csslayer.info/wordpress/linux/rfc-a-new-solution-to-input-method-keyboard- layout/ And that is complex, and not be able to clearly defined with simple configuration file or some combo box option. Compositor should really give this kind of freedom to input method daemon. It doesn't expose extra security issue if you think about it. kwin would just works like a router that talks with input method via wayland protocol. And kwin will not talk with some random input method daemon that it unprivileged.
Re: Complex text input in Plasma
Am 2017-04-07 07:56, schrieb Takao Fujiwara: Due to that: no chance for IM controlling part of our stack. We control the IM. Probably I think this way would not work with IBus. Each IBus IME are called by IBus dbus method. You hardly ask each IME maintainer to change the protocol. IBus daemon would be the controller of IBUs IMEs. I think I need to describe in a better way how I envision the future architecture. I want to have the IM daemon merged into the wayland compositor. So KWin would talk directly with the IM through library calls and the composed text would be sent from KWin to the application through the Wayland text-input protocol. I want to eliminate the IPC between applications and IM daemon. If we are going to touch this code we can strive for the best solution and not keep stuck on the approach used on X11. And yes that might need adjusting the IMEs. But they need to be adjusted anyway. We wouldn't have this discussion if it would just work on Wayland. Cheers Martin
Re: Complex text input in Plasma
On Thursday, 6 April 2017 19:16:14 CEST Eike Hein wrote: > Hi, > > In the aftermath of D5301, Martin asked to compile a document on the > requirements for complex text input in Plasma, especially with the > opportunities provided by the Wayland transition. It makes sense to > share this document with all of you, to share the knowledge more widely :) I worked a lot recently on updating the IME support in Slackware so I'll add a couple points from my experience with that as well as from a packager perspective. I evaluated both ibus and fcitx and found that, much like Eike already explained in his initial mail, initial setup is less than ideal. This is even more so the case on Slackware, since we do not have distro-specific tooling for the user to select a language or input method. = Base setup = I'll describe the raw setup of both ibus and fcitx on Slackware, as well as some of the "improvements" I made to make this a bit simpler for the user (I guess one could call that distro-specific tooling as well). a) ibus After login you execute "ibus-setup" which checks whether the ibus daemon is running and starts it otherwise before showing its configuration dialog. Once finished with the configuration it asks to add a couple entries to the user's .bashrc file: export XMODIFIERS=@im=ibus export GTK_IM_MODULE=ibus export QT_IM_MODULE=ibus After this the user still has to manually configure Qt4 to use "ibus" as the default input method (via qtconfig), as well as add some way to start ibus- daemon automatically on login. b) fcitx fcitx is a bit userfriendlier in that it already provides xdg autostart support (though most distros disable this). So if fcitx is installed it will automatically run on login, even if the user doesn't need it. It still needs similar steps though as ibus. You configure your input methods via fcitx' configuration tool and add similar entries to .bashrc: export XMODIFIERS="@im=fcitx" export XIM=fcitx export XIM_PROGRAM=fcitx export GTK_IM_MODULE=fcitx export QT_IM_MODULE=fcitx You also again have to set Qt4's default input method to "fcitx". c) Modifications This plain setup leaves us with some "papercuts" though. fcitx's autostart doesn't really play well with ibus. This is, if ibus is configured and used, and both ibus and fcitx are installed, both would be running. Worst case scenario this means the user has 3 input method indicators active in the system tray (ibus, fcitx and xkb) I "fixed" this by making fcitx' autostart dependent on the user's .bashrc entries, so it only starts if fcitx is configured as input method using the before mentioned entries. I added a similar autostart script for ibus, with the effect that now at least only the one configured for the user is running and the user doesn't have to worry about the autostart part of the setup. = Distro comparison = While researching this initial setup routine I looked around at what other distros are doing, only to find that mostly all of them do something different. - Debain uses im-switch, which seems to be this: https://github.com/pkg-ime/ im-switch - Opensuse uses im-config. I couldn't really find the source for that, but it looks to work similar to Debian's im-switch - Fedora uses some kind of mechanism with xinputrc and xinput.d in order to load input methods during startup. Couldn't really find too much documentation about this either though :-/ = Future = Generally I'm very happy with this effort and I'm looking forward to an improved input method integration in Plasma 5. However, putting my distro-hat on, I really think that this is a problem that goes beyond Plasma 5. I understand that desktop specific solutions need to be found, but the same issue exists on other desktop environments and it would be a shame if they have to reinvent the wheel on their end again. Eike mentioned fragmentation in the input method field and I'd say this definitely adds to that fragmentation. If whatever solution is eventually picked leaves the door open for other desktop environments to provide the same kind of integration I think this would definitely be appreciated :) Grs, Heinz signature.asc Description: This is a digitally signed message part.
Re: Complex text input in Plasma
On 04/07/17 14:13, Martin Gräßlin-san wrote: Am 2017-04-06 22:18, schrieb Eike Hein: On 04/07/2017 04:58 AM, Weng Xuetian wrote: gnome-shell's kimpanel equilvalent stuff is bundled in gnome-shell, which make it be able to access the windows position. Input method server send the offset received from client (which is not global), and the gnome-shell move the panel to the (offset+current window location) This seems to be roughly what Martin wants to do as well, except I think he was probably envisioning the window being part of KWin somehow. Yes GNOME Shell (Wayland compositor) and KWin (Wayland compositor) are the same layer in the stack. My understanding is that it might be necessary to allow the IME to provide the UI however, since one-list-popup-fits-all may not be true across writing systems. 4. Let input method server controls the keyboard layout (which layout to use) Keyboard layout is nothing special comparing with input method. Nowadays, modern input method framework are trying to take over all this stuff. This is essential for users to get best experience if they use multiple input method. Because there's a concept called "input context", which is not essentially one-to-one map to the window. I saw this one coming and it's the reason I originally spoke up in https://phabricator.kde.org/D5301 - there's ongoing work on bringing keyboard layout switching policies to kwin_wayland right now, in a system that isn't integrated with the input method systems. This shows the need for coordination. Important note here: IM is not able to control the keyboard layout. That must be in the Wayland compositor. A downside with the "input method always on" approach can be performance: We don't want to send events through layers of IPC for input scenarios we don't want to. This is just another reason to have it in KWin which would completely eliminate the IPC. Also from a security perspective way better. There is no chance for an integration of a system where every key event is sent through an IPC. That wouldn't pass my security review. This reads on things like the work in D5301. Scenarios: * If the input method is always on, then there maybe doesn't need to be a fallback to complex, redundant policy systems in KWin. * If the input method continues to be an optional add-on, then KWin still needs those policy systems. * The input method could be rearchitected to drive KWin's built-in policy systems via public API. I dislike all three options. The base keyboard layout handling must be in KWin. KWin has to control the IM stack, not the other way around. I'm not giving away control over such fundamental aspects of the stack any more. I want that these things function correctly and I'm not trusting 3rd party software to do the right things. KWin can do the right thing in all situations. KWin is the only element in the stack having a global view: is the screen locked, is virtual keyboard on, is a fullscreen effect in place, is a popup menu open, is a game being played? Only KWin knows that. Only KWin can ensure that the right thing is done all the time. Due to that: no chance for IM controlling part of our stack. We control the IM. Probably I think this way would not work with IBus. Each IBus IME are called by IBus dbus method. You hardly ask each IME maintainer to change the protocol. IBus daemon would be the controller of IBUs IMEs. Thanks, Fujiwara Cheers Martin
Re: Complex text input in Plasma
Am 2017-04-06 21:50, schrieb Eike Hein: Weng as both a fcitx and Plasma developer is at a critical intersection here: You and Martin will need to work together on how fcitx can fit into a kwin_wayland world. may I suggest a name change here: please replace Martin by Eike :-) I think you are a way better fit to work on it as I'm just not that familiar with the stack and to me input is still something new. I'm happy to have interested devs to take over this part in KWin. Cheers Martin
Re: Complex text input in Plasma
Am 2017-04-06 22:18, schrieb Eike Hein: On 04/07/2017 04:58 AM, Weng Xuetian wrote: gnome-shell's kimpanel equilvalent stuff is bundled in gnome-shell, which make it be able to access the windows position. Input method server send the offset received from client (which is not global), and the gnome-shell move the panel to the (offset+current window location) This seems to be roughly what Martin wants to do as well, except I think he was probably envisioning the window being part of KWin somehow. Yes GNOME Shell (Wayland compositor) and KWin (Wayland compositor) are the same layer in the stack. My understanding is that it might be necessary to allow the IME to provide the UI however, since one-list-popup-fits-all may not be true across writing systems. 4. Let input method server controls the keyboard layout (which layout to use) Keyboard layout is nothing special comparing with input method. Nowadays, modern input method framework are trying to take over all this stuff. This is essential for users to get best experience if they use multiple input method. Because there's a concept called "input context", which is not essentially one-to-one map to the window. I saw this one coming and it's the reason I originally spoke up in https://phabricator.kde.org/D5301 - there's ongoing work on bringing keyboard layout switching policies to kwin_wayland right now, in a system that isn't integrated with the input method systems. This shows the need for coordination. Important note here: IM is not able to control the keyboard layout. That must be in the Wayland compositor. A downside with the "input method always on" approach can be performance: We don't want to send events through layers of IPC for input scenarios we don't want to. This is just another reason to have it in KWin which would completely eliminate the IPC. Also from a security perspective way better. There is no chance for an integration of a system where every key event is sent through an IPC. That wouldn't pass my security review. This reads on things like the work in D5301. Scenarios: * If the input method is always on, then there maybe doesn't need to be a fallback to complex, redundant policy systems in KWin. * If the input method continues to be an optional add-on, then KWin still needs those policy systems. * The input method could be rearchitected to drive KWin's built-in policy systems via public API. I dislike all three options. The base keyboard layout handling must be in KWin. KWin has to control the IM stack, not the other way around. I'm not giving away control over such fundamental aspects of the stack any more. I want that these things function correctly and I'm not trusting 3rd party software to do the right things. KWin can do the right thing in all situations. KWin is the only element in the stack having a global view: is the screen locked, is virtual keyboard on, is a fullscreen effect in place, is a popup menu open, is a game being played? Only KWin knows that. Only KWin can ensure that the right thing is done all the time. Due to that: no chance for IM controlling part of our stack. We control the IM. Cheers Martin
Re: Complex text input in Plasma
IBus emoji typing is updated day by day. IBus 1.5.15 moved the emoji function in IBus XKB engine to IBus panel, which means the UI is now available from pane GUI menu and the shortcut key can be customizable with ibus-setup: https://desktopi18n.wordpress.com/2017/03/15/ibus-1-5-15-is-released/ Also 'ibus emoji' CLI can launch the same GUI and it does not require to run ibus-daemon. The emoji tying has two goals: - available with an extended lookup window likes an input method's lookup window without mouse operations for input method users - available from panel menu by mouse only for GUI users Thanks, Fujiwara On 04/07/17 05:18, Eike Hein-san wrote: On 04/07/2017 04:58 AM, Weng Xuetian wrote: gnome-shell's kimpanel equilvalent stuff is bundled in gnome-shell, which make it be able to access the windows position. Input method server send the offset received from client (which is not global), and the gnome-shell move the panel to the (offset+current window location) This seems to be roughly what Martin wants to do as well, except I think he was probably envisioning the window being part of KWin somehow. My understanding is that it might be necessary to allow the IME to provide the UI however, since one-list-popup-fits-all may not be true across writing systems. 4. Let input method server controls the keyboard layout (which layout to use) Keyboard layout is nothing special comparing with input method. Nowadays, modern input method framework are trying to take over all this stuff. This is essential for users to get best experience if they use multiple input method. Because there's a concept called "input context", which is not essentially one-to-one map to the window. I saw this one coming and it's the reason I originally spoke up in https://phabricator.kde.org/D5301 - there's ongoing work on bringing keyboard layout switching policies to kwin_wayland right now, in a system that isn't integrated with the input method systems. This shows the need for coordination. Taking a step back, there's a couple of ways to go on the system architecture. Right now, an input method system is kind of an "after market add-on" - it's something you install only if you need to, and if you do, it replaces parts of your system with its own stuff. Thoughts: a) The "replaces part of your system with its own stuff" is what gets us the unintegrated config mess, where your System Settings becomes useless if the input method is around. b) Historically a input method is used by non-Latin-based language users, but with Emoji input and things like word completion, this is changing. I think a and b together mean that an input method is now better positioned as a core dependency of the system, not a special case. A downside with the "input method always on" approach can be performance: We don't want to send events through layers of IPC for input scenarios we don't want to. This reads on things like the work in D5301. Scenarios: * If the input method is always on, then there maybe doesn't need to be a fallback to complex, redundant policy systems in KWin. * If the input method continues to be an optional add-on, then KWin still needs those policy systems. * The input method could be rearchitected to drive KWin's built-in policy systems via public API. Cheers, Eike
Re: Complex text input in Plasma
On 04/07/2017 04:58 AM, Weng Xuetian wrote: > gnome-shell's kimpanel equilvalent stuff is bundled in gnome-shell, which > make > it be able to access the windows position. Input method server send the > offset > received from client (which is not global), and the gnome-shell move the > panel > to the (offset+current window location) This seems to be roughly what Martin wants to do as well, except I think he was probably envisioning the window being part of KWin somehow. My understanding is that it might be necessary to allow the IME to provide the UI however, since one-list-popup-fits-all may not be true across writing systems. > 4. Let input method server controls the keyboard layout (which layout to use) > Keyboard layout is nothing special comparing with input method. Nowadays, > modern input method framework are trying to take over all this stuff. This is > essential for users to get best experience if they use multiple input method. > Because there's a concept called "input context", which is not essentially > one-to-one map to the window. I saw this one coming and it's the reason I originally spoke up in https://phabricator.kde.org/D5301 - there's ongoing work on bringing keyboard layout switching policies to kwin_wayland right now, in a system that isn't integrated with the input method systems. This shows the need for coordination. Taking a step back, there's a couple of ways to go on the system architecture. Right now, an input method system is kind of an "after market add-on" - it's something you install only if you need to, and if you do, it replaces parts of your system with its own stuff. Thoughts: a) The "replaces part of your system with its own stuff" is what gets us the unintegrated config mess, where your System Settings becomes useless if the input method is around. b) Historically a input method is used by non-Latin-based language users, but with Emoji input and things like word completion, this is changing. I think a and b together mean that an input method is now better positioned as a core dependency of the system, not a special case. A downside with the "input method always on" approach can be performance: We don't want to send events through layers of IPC for input scenarios we don't want to. This reads on things like the work in D5301. Scenarios: * If the input method is always on, then there maybe doesn't need to be a fallback to complex, redundant policy systems in KWin. * If the input method continues to be an optional add-on, then KWin still needs those policy systems. * The input method could be rearchitected to drive KWin's built-in policy systems via public API. Cheers, Eike
Re: Complex text input in Plasma
On Thursday, 6 April 2017 10:27:40 PDT，Eike Hein wrote： > > = Screenshots = > > Here's an example of the emoji IME mentioned earlier: > > https://fedoramagazine.org/using-favorite-emoji-fedora-25/ > > > Cheers, > Eike As a side note, I used to implment a ugly imchooser for KDE around KDE4 time, which helps to select which input method https://www.linux-apps.com/content/show.php/kcm+imchooser?content=146776 If we don't want to go the GNOME way (solely select one input method frameworks to work with), I guess one thing we can do is to add an additonal section in "default application" kcm, and provides necesary profile, so kwin will also be able to learn which im to start, and what need to be set up. the profile may also provide a "none" which fallback to the old switcher.
Re: Complex text input in Plasma
On Thursday, 6 April 2017 10:16:14 PDT，Eike Hein wrote： > Hi, > > In the aftermath of D5301, Martin asked to compile a document on the > requirements for complex text input in Plasma, especially with the > opportunities provided by the Wayland transition. It makes sense to > share this document with all of you, to > > > = I. What Input Methods do = > > Basically, with simple text input (and skipping a few stack steps in > this explanation), the user presses a key on the keyboard, it's > interpreted according to the active layout, and results in a symbol > on screen (modulo complicated ideas like dead keys and the Compose > key, which also enter into input method territory as we'll see). > > With some writing system, this isn't enough. They require multiple > key presses or pick-and-choose selection steps to produce a symbol. > The general idea is to disconnect the input from the output and > introduce a conversion inbetween that's more complex than a mere > keyboard layout. The conversion itself may have complex UI like > temporary feedback in the textfield, popups positioned relative > to the text field, or state visualization in shell chrome. > > There are other ways to access input methods than just the keyboard > these days, too. I'll briefly mention one of them later. > > > = II. Practical examples of Input Method use = > > a) Korean > > Korean is written using an alphabet similar to German. It has about > the same number of letters, with each letter representing a vowel, > a consonant, or a diphtong. > > On computers, each letter has its own key on the keyboard. As with > German, there are multiple keyboard layouts for the Korean alphabet, > although one is dominant and considered standard by far. > > Unlike the German alphabet however, letters in the Korean alphabet > are grouped together into (morpho-)syllabic blocks when written. > Each block must start with a consonant and contain a vowel or > diphtong. It may optionally end in one or two consonants. > > Here are some letters and their corresponding Latin latterns (the > sounds they most closely correspond to - theeir locations in the > keyboard layout do not match QWERTY/QWERTZ): > > ㅎ h > ㅏ a > ㄴ n > > In linear order these form the syllable "han". When written > properly, these are groupedtogether: > > 한 han > > In the UI, when pressing the keyboard keys for each letter, the > text field contents cycle through these stages: > > ㅎ > 하 > 한 > > As you can see, the existing text is replaced two times, making > the operation stateful. The Input Method Engine generates complex > events with text payload, state hints, even formatting hints (some > IMEs use text color, e.g. color inversion, to communicate state > such as "this input is not finalized yet") that are delivered to > the application. In Qt, a plugin corresponding to the Input Method > (e.g. ibus or fcitx) translates these events into QInputMethodEvent > objects that are delivered to widgets and processed there. > > The rules of the Korean alphabet additionally have some > implications for cursor movement/behavior. E.g. because it's not > allowed to start a block with two consonants, and because the number > of vowels and ending consonants is limited, as keys are pressed a > block might implicitly finish composing and the cursor moves on. > > b) Chinese > > There are many different strategies for inputting Chinese > characters. The most common actually makes use of the Latin > alphabet, and keyboard layouts for the Latin alphabet. Chinese > characters have assigned sound values (i.e., how a human actually > pronounces a character), and there are rule systems to transcribe > these using the Latin alphabet, e.g. Pinyin. > > As users write some Pinyin using Latin characters, a selection > popup will offer a list of Chinese characters matching the input. > The user picks one and it's inserted into the text field. > > Chinese input methods try to be very smart in what characters > they offer, taking preceding input, common phrasings, etc. into > account, making them highly stateful things. > > c) Modes, Input Method overlap > > Korean used to be written using Chinese characters (this simple > statement is the tip of a large iceberg of complicated history > and rulesets :), and especially Korean Academic writing still > makes use of them. Korean input methods wherefore tend to also > offer the ability to type Chinese characters, with a mode toggle > between them. > > Korean input methods also usually have a Hangul (the Korean > alphabet) vs. Latin mode toggle. > > Chinese input methods tend to have mode toggles for things l
Re: Complex text input in Plasma
On 04/07/2017 04:21 AM, Martin Gräßlin wrote: > that's actually also a feature of the keyboard daemon we have on X11. > Only on wayland KWin takes care of it. Aye, I kind of used KWin as an insert for our whole amalgamation of shell infra there. > Ideally we would build up on this. We must not (!) specify an input > plugin for Qt apps as that would break the virtual keyboard. And even if > others give up on convergence I still think it's important to support it > and have it as a target. > > So for input methods KWin would have to provide the UI interaction. Of > course I do not want to rebuild such a stack but would prefer to be able > to hook into an existing - I hope that's possible. Maybe a similar > approach as to how virtual keyboard works: kwin just emulates to the > qtvirtualkeyboard like if it were in application. So maybe KWin could > load one of the plugins and interact with them as a proxy for the > application and just forward the result through the text input interface. It's super imperative that we work together with the input method community, then. This space only has relatively few active developers unfortunately, and the developer base is already spread thin over multiple competing stacks. Additionally, much of the development activity doesn't even happen in the core, but at the fringes, in the form of IME plugins that are often developed by local communities in isolation from the core - due to language barrier effects. If KWin were to introduce another entirely new IME plugin API to target, we'd further fragment the ecosystem, require IME developers to duplicate much work, and potentially miss out on work that continues to be done against other APIs. (It's important to point out that IMEs are not "done". Emoji IMEs are a recent development, and things how "How best to type Chinese into a computer?" are still open questions that see active experimentation, despite established solutions. Last but not least, not every popular writing system is covered by free systems already: There are still entire populations that basically can't type text on a free desktop system. Bottom line, an important requirement is future extensibility and trying to funnel work done our way.) This fragmentation has a big burden all around - most importantly on users, because tons of IME plugins are single-person projects with quality issues. As good platform shepards we should recognize these systemic problems and try to heal them. The goal being "bring high-quality text input to as many people as possible" and any steps measured by that yardstick. Fortunately the way the stack has stratified, we do control the UI frontend (or rather we offer one of our own at least, as per my longer overview). Input Method Panel's popup stuff may wander into KWin along with other interaction bits - we're free to do that. Weng as both a fcitx and Plasma developer is at a critical intersection here: You and Martin will need to work together on how fcitx can fit into a kwin_wayland world. We should also look beyond our borders and look at how ibus on Gnome/Wayland is being done. > What I could imagine is that we start with emoji support. Might sound > weird but is a simplified use case to just "play" with it and > experiment. We could setup a global shortcut (like the one in GNOME), > pop up a UI with the emoji selection and go through the text input > protocol to send the emoji into the application. We could definitely use that to prototype interaction bits and stub out the backend part for now. As mentioned ideally the business logic for it would eventually come from a fcitx/ibus plugin. Cheers, Eike
Re: Complex text input in Plasma
Am 2017-04-06 19:16, schrieb Eike Hein: Hi, In the aftermath of D5301, Martin asked to compile a document on the requirements for complex text input in Plasma, especially with the opportunities provided by the Wayland transition. It makes sense to share this document with all of you, to Thanks for the nice write down. * KWin is smart enough to do cool things like switch keyboard layouts automatically per virtual desktop or even window that's actually also a feature of the keyboard daemon we have on X11. Only on wayland KWin takes care of it. = The situation on Wayland = Currently, only ibus works on Wayland. Input method popups are not positioned correctly, but otherwise things work. The situation is just as good or bad as on X11. Sharing a bit more about state of Wayland: There is the text-input protocol which KWin supports and uses for virtual keyboard. This means we already have the possibility to do things like text input, text prediction, cursor moving, styling hints (not yet implemented as virt keyboard doesn't support it), etc. etc. out of the box with Qt applications. We know whether the application supports it. We know the position of the text field and much more. For those interested: relevant code is in KWin/virtualkeyboard.cpp Ideally we would build up on this. We must not (!) specify an input plugin for Qt apps as that would break the virtual keyboard. And even if others give up on convergence I still think it's important to support it and have it as a target. So for input methods KWin would have to provide the UI interaction. Of course I do not want to rebuild such a stack but would prefer to be able to hook into an existing - I hope that's possible. Maybe a similar approach as to how virtual keyboard works: kwin just emulates to the qtvirtualkeyboard like if it were in application. So maybe KWin could load one of the plugins and interact with them as a proxy for the application and just forward the result through the text input interface. What I could imagine is that we start with emoji support. Might sound weird but is a simplified use case to just "play" with it and experiment. We could setup a global shortcut (like the one in GNOME), pop up a UI with the emoji selection and go through the text input protocol to send the emoji into the application. Unfortunately text input in Wayland is weird as there are two competing protocol versions and Qt has one of them. There is the zwp_text_input_v1 which is part of Wayland-protocols and zwp_text_input_v2 which is part of Qt. It is not upstreamed, but uses upstream naming conventions. Which will result in a horrible mess once upstream introduces a zwp_text_input_v2. KWin supports both versions at the same time, but it makes everything more complex and we currently can only assume that it works correctly with Qt apps. The GTK version we support on Wayland neither supports v1 nor v2. For the newer GTK versions which we do not support yet I do not know. Also it leaves out Xwayland application but there a bridge to one of the existing X11 solutions is thinkable. Cheers Martin
Re: Complex text input in Plasma
> = Screenshots = Here's an example of the emoji IME mentioned earlier: https://fedoramagazine.org/using-favorite-emoji-fedora-25/ Cheers, Eike
Re: Complex text input in Plasma
On 04/07/2017 02:16 AM, Eike Hein wrote: > > Hi, > > In the aftermath of D5301, Martin asked to compile a document on the > requirements for complex text input in Plasma, especially with the > opportunities provided by the Wayland transition. It makes sense to > share this document with all of you, to *to share the knowledge more widely :) Cheers, Eike
Complex text input in Plasma
Hi, In the aftermath of D5301, Martin asked to compile a document on the requirements for complex text input in Plasma, especially with the opportunities provided by the Wayland transition. It makes sense to share this document with all of you, to = I. What Input Methods do = Basically, with simple text input (and skipping a few stack steps in this explanation), the user presses a key on the keyboard, it's interpreted according to the active layout, and results in a symbol on screen (modulo complicated ideas like dead keys and the Compose key, which also enter into input method territory as we'll see). With some writing system, this isn't enough. They require multiple key presses or pick-and-choose selection steps to produce a symbol. The general idea is to disconnect the input from the output and introduce a conversion inbetween that's more complex than a mere keyboard layout. The conversion itself may have complex UI like temporary feedback in the textfield, popups positioned relative to the text field, or state visualization in shell chrome. There are other ways to access input methods than just the keyboard these days, too. I'll briefly mention one of them later. = II. Practical examples of Input Method use = a) Korean Korean is written using an alphabet similar to German. It has about the same number of letters, with each letter representing a vowel, a consonant, or a diphtong. On computers, each letter has its own key on the keyboard. As with German, there are multiple keyboard layouts for the Korean alphabet, although one is dominant and considered standard by far. Unlike the German alphabet however, letters in the Korean alphabet are grouped together into (morpho-)syllabic blocks when written. Each block must start with a consonant and contain a vowel or diphtong. It may optionally end in one or two consonants. Here are some letters and their corresponding Latin latterns (the sounds they most closely correspond to - theeir locations in the keyboard layout do not match QWERTY/QWERTZ): ㅎ h ㅏ a ㄴ n In linear order these form the syllable "han". When written properly, these are groupedtogether: 한 han In the UI, when pressing the keyboard keys for each letter, the text field contents cycle through these stages: ㅎ 하 한 As you can see, the existing text is replaced two times, making the operation stateful. The Input Method Engine generates complex events with text payload, state hints, even formatting hints (some IMEs use text color, e.g. color inversion, to communicate state such as "this input is not finalized yet") that are delivered to the application. In Qt, a plugin corresponding to the Input Method (e.g. ibus or fcitx) translates these events into QInputMethodEvent objects that are delivered to widgets and processed there. The rules of the Korean alphabet additionally have some implications for cursor movement/behavior. E.g. because it's not allowed to start a block with two consonants, and because the number of vowels and ending consonants is limited, as keys are pressed a block might implicitly finish composing and the cursor moves on. b) Chinese There are many different strategies for inputting Chinese characters. The most common actually makes use of the Latin alphabet, and keyboard layouts for the Latin alphabet. Chinese characters have assigned sound values (i.e., how a human actually pronounces a character), and there are rule systems to transcribe these using the Latin alphabet, e.g. Pinyin. As users write some Pinyin using Latin characters, a selection popup will offer a list of Chinese characters matching the input. The user picks one and it's inserted into the text field. Chinese input methods try to be very smart in what characters they offer, taking preceding input, common phrasings, etc. into account, making them highly stateful things. c) Modes, Input Method overlap Korean used to be written using Chinese characters (this simple statement is the tip of a large iceberg of complicated history and rulesets :), and especially Korean Academic writing still makes use of them. Korean input methods wherefore tend to also offer the ability to type Chinese characters, with a mode toggle between them. Korean input methods also usually have a Hangul (the Korean alphabet) vs. Latin mode toggle. Chinese input methods tend to have mode toggles for things like chosing between half-width and full-width characters, i.e. also offer control over typography. Japanese is a mashup of all of these things, with Japanese users typing in two Japanese-specific syllabaries, Chinese characters and Latin during a typical session. = Other uses for input methods = The ever more popular Emoji character set may turn writers of languages which typically do not require an input method into input method users. On Fedora/Gnome systems, pressing Ctrl+Shift+e inserts a @ character into the text field, and typing a string such as "heart" will select among su
