Re: Future handling of complex RGB devices on Linux v2
Hi Am 22.02.24 um 18:38 schrieb Pavel Machek: Hi! so after more feedback from the OpenRGB maintainers I came up with an even more generic proposal: https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 evaluate-set-command ioctl taking: { enum command /* one of supported_commands */ union data { char raw[3072], { }, Yeah, so ... this is not a interface. This is a backdoor to pass arbitrary data. That's not going to fly. For keyboards, we don't need complete new interface; we reasonable extensions over existing display APIs -- keyboards are clearly 2D. I suppose they could be seen as *a* display, but if you are referring to DRM KMS UAPI, then no, I don't see that fitting at all: So -- we already have very similar displays in drivers/auxdisplay. drivers/auxdisplay/cfag12864b.c is 128x64 display, 1-bit display for example. Auxdisplay can be anything for everyone. It appears to be the rest that didn't clearly fit elsewhere. The core interface seems to be a custom char device. The fbdev code in cfag12864b is not representative. - the "pixel grid" is not orthogonal, it's not a rectangle, and it might not be a grid at all It is quite close to orthogonal. I'd suggest simply pretending it is orthogonal grid with some pixels missing :-). We already have cellphone displays with rounded corners and holes in them, so I suspect handling of missing pixels will be neccessary anyway. - Timings and video modes? DRM KMS has always been somewhat awkward for display devices that do not have an inherent scanout cycle and timings totally depend on the amount of pixels updated at a time (FB_DAMAGE_CLIPS), e.g. USB displays (not USB-C DP alt mode). They do work, but they are very different from the usual hardware involved with KMS, require special consideration in userspace, and they still are actual displays while what we're talking about here are not. As you say, there are other displays with similar problems. - KMS has no concept of programmed autonomous animations, and likely never will. They are not useful with actual displays. Yep. We need some kind of extension here, and this is likely be quite vendor-specific, as animations will differ between vendors. I guess "please play pattern xyzzy with parametrs 3 and 5" might be enough for this. The litmus test for DRM and fbdev is something like "would the user run the console, desktop, or any other meaningful output in this display". That is also what userspace (e.g., X, Wayland, gfx terminals) expects: a display to show the user's main output. Keyboard LEDs don't fit here. Best regards Thomas - Userspace will try to light up KMS outputs automatically and extend the traditional desktop there. This was already a problem for head-mounted displays (HMD) where it made no sense. That was worked around with an in-kernel list of HMDs and some KMS property quirking. This will need fixing for cfag12864b.c, no? Perhaps userspace should simply ignore anything smaller than 240x160 or something... Modern KMS UAPI very much aims to be a generic UAPI that abstracts display devices. It already breaks down a little for things like USB displays and virtual machines (e.g. qemu, vmware, especially with remote viewers), which I find unfortunate. With HMDs the genericity breaks down in other ways, but I'd claim HMDs are a better fit still than full-featured VM virtual displays (cursor plane hijacking). With non-displays like keyboards the genericity would be completely lost, as they won't work at all the same way as displays. You cannot even show proper images there, only coarse light patterns *IF* you actually know the pixel layout. But the pixel layout is(?) hardware-specific which is the opposite of generic. While you could dress keyboard lights etc. up with DRM KMS UAPI, the userspace would have to be written from scratch for them, and you somehow need to make existing KMS userspace to never touch those devices. What's the point of using DRM KMS UAPI in the first place, then? Well, at least we have good UAPI to work with. Other options were 100 files in /sys/class/leds, pretending it is a linear array of leds, just passing raw data around, and pretending it is grid of RGB888 data. Anyway, there are devices such as these: (8x8 LED display). https://www.laskakit.cz/8x8-led-matice-s-max7219-3mm-cervena/ We should think about what interface we want for these, and then I believe we should make RGB keyboards use something similar. Best regards, Pavel -- -- Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Frankenstrasse 146, 90461 Nuernberg, Germany GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman HRB 36809 (AG Nuernberg)
Re: Future handling of complex RGB devices on Linux v2
On Thu, 22 Feb 2024 18:38:16 +0100 Pavel Machek wrote: > Hi! > > > > > so after more feedback from the OpenRGB maintainers I came up with an > > > > even > > > > more generic proposal: > > > > https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 > > > > > > > > > > > >evaluate-set-command ioctl taking: > > > > >{ > > > > > enum command /* one of supported_commands */ > > > > > union data > > > > > { > > > > > char raw[3072], > > > > > { > > > > > > > > > > }, > > > > > > Yeah, so ... this is not a interface. This is a backdoor to pass > > > arbitrary data. That's not going to fly. > > > > > > For keyboards, we don't need complete new interface; we reasonable > > > extensions over existing display APIs -- keyboards are clearly 2D. > > > > I suppose they could be seen as *a* display, but if you are referring > > to DRM KMS UAPI, then no, I don't see that fitting at all: > > So -- we already have very similar displays in > drivers/auxdisplay. drivers/auxdisplay/cfag12864b.c is 128x64 display, > 1-bit display for example. Auxdisplay are not exposed as DRM KMS device, or are they? I don't see cfag12864b.c using any DRM calls. DRM drivers are under drivers/gpu/drm. I know nothing about auxdisplay. If it's fbdev UAPI, then I don't know - people have been trying to kill it for years, and basing anything on it seems like a dead idea. Or maybe it's ok for extremely small displays where there is no display controller to speak of, and definitely no use ever for dmabuf? Where CPU banging bits of raw pixels is enough, and no desktop would ever want to touch them. But I'm not talking about fbdev either, I'm talking about DRM KMS. > > > - the "pixel grid" is not orthogonal, it's not a rectangle, and it > > might not be a grid at all > > It is quite close to orthogonal. I'd suggest simply pretending it is > orthogonal grid with some pixels missing :-). We already have > cellphone displays with rounded corners and holes in them, so I > suspect handling of missing pixels will be neccessary anyway. Yes, but I do not agree on the similarity at all, nor that it would be "close to orthogonal" by simply looking at my keyboard. Hans de Goede iterated on this much better than I. > > - Timings and video modes? DRM KMS has always been somewhat awkward for > > display devices that do not have an inherent scanout cycle and timings > > totally depend on the amount of pixels updated at a time > > (FB_DAMAGE_CLIPS), e.g. USB displays (not USB-C DP alt mode). > > They do work, but they are very different from the usual hardware > > involved with KMS, require special consideration in userspace, and > > they still are actual displays while what we're talking about here > > are not. > > As you say, there are other displays with similar problems. That's no justification to add even more problems. > > - KMS has no concept of programmed autonomous animations, and likely > > never will. They are not useful with actual displays. > > Yep. We need some kind of extension here, and this is likely be quite > vendor-specific, as animations will differ between vendors. I guess > "please play pattern xyzzy with parametrs 3 and 5" might be enough for this. Right. I very much believe that is not going to fly in DRM KMS. > > - Userspace will try to light up KMS outputs automatically and extend > > the traditional desktop there. This was already a problem for > > head-mounted displays (HMD) where it made no sense. That was worked > > around with an in-kernel list of HMDs and some KMS property > > quirking. > > This will need fixing for cfag12864b.c, no? Perhaps userspace should > simply ignore anything smaller than 240x160 or something... Yes, a size limit would make sense in desktop usage. > > Modern KMS UAPI very much aims to be a generic UAPI that abstracts > > display devices. It already breaks down a little for things like USB > > displays and virtual machines (e.g. qemu, vmware, especially with > > remote viewers), which I find unfortunate. With HMDs the genericity > > breaks down in other ways, but I'd claim HMDs are a better fit still > > than full-featured VM virtual displays (cursor plane hijacking). With > > non-displays like keyboards the genericity would be completely lost, as > > they won't work at all the same way as displays. You cannot even show > > proper images there, only coarse light patterns *IF* you actually know > > the pixel layout. But the pixel layout is(?) hardware-specific which is > > the opposite of generic. > > > > While you could dress keyboard lights etc. up with DRM KMS UAPI, the > > userspace would have to be written from scratch for them, and you > > somehow need to make existing KMS userspace to never touch those > > devices. What's the point of using DRM KMS UAPI in the first place, > > then? > > Well, at least we have good UAPI to work with. ..Where the great
Re: Future handling of complex RGB devices on Linux v2
Hi, Am 21.02.24 um 23:17 schrieb Pavel Machek: Hi! so after more feedback from the OpenRGB maintainers I came up with an even more generic proposal: https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 evaluate-set-command ioctl taking: { enum command /* one of supported_commands */ union data { char raw[3072], { }, Yeah, so ... this is not a interface. This is a backdoor to pass arbitrary data. That's not going to fly. For keyboards, we don't need complete new interface; we reasonable extensions over existing display APIs -- keyboards are clearly 2D. Maybe we should look on this from a users perspective: Running custom Animation (i.e. where treating it as a display would be helpfull) is only >one< of the ways a user might want to use the keyboard backlight. Equally viable are for example: - Having it mono colored. - Playing a hardware effect - Playing a hardware effect on one side of the keyboard while having the other side of the keyboard playing a custom animation (As I learned from the OpenRGB maintainers: There are keyboards which allow this) There is no reason to define a custom animation as the default and then just bolt the other options on top as it might not even be possible for some devices where the firmware is plainly to slow for custom animations. Also I still think a 2*2, 1*3 or even 1*1 matrix is not a display, coming back aground to the earlier point where I want to implement this for keyboard first, but this discussion is also thought to find a way that works for all complex RGB devices. And I don't think it is a good idea find a way that works for keyboards and then introduce again something completely new for other device categories. Best regards, Pavel Best regards, Werner
Re: Future handling of complex RGB devices on Linux v2
Hi! > For all these reasons the display analogy really is a bit fit for these > keyboards > we tried to come up with a universal coordinate system for these at the > beginning > of the thread and we failed ... I quite liked the coordinate system proposal. I can propose this: Vendor maps the keyboard lights to a grid. That would be something 16x8 for thinkpad X220. Then we provide functionality to query "is a working pixel there" and "what kind of key is at this pixel" -- I guess we can use input keycodes for that. Multiple pixels can map to one keycode. (And then we make best effort to map normal keyboards into similar grids). Best regards, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. signature.asc Description: PGP signature
Re: Future handling of complex RGB devices on Linux v2
Hi! > > To be honest, I think the kernel shouldn't include too much high-level > > complexity. If there is a desire to implement a generic display device on > > top of the RGB device, this should be a configurable service running in > > user space. The kernel should provide an interface to expose this emulated > > display as a "real" display to applications - unless this can also be done > > entirely in user space in a generic way. > > We really need to stop seeing per key addressable RGB keyboards as displays: > > 1. Some "pixels" are non square > 2. Not all "pixels" have the same width-height ratio They are quite close to square usually. > 3. Not all rows have the same amount of pixels True for cellphone displays, too. Rounded corners. > 4. There are holes in the rows like between the enter key and then numpad True for cellphone displays, too. Hole for camera. > 5. Some "pixels" have multiple LEDs beneath them. These might be addressable >per LEDs are the sub-pixels ? What about a 2 key wide backspace key vs >the 1 key wide + another key (some non US layouts) in place of the > backspace? >This will be "2 pixels" in some layout and 1 pixel with maybe / maybe-not >2 subpixels where the sub-pixels may/may not be individually addressable ? Treat those "sub pixels" as pixels. They will be in same matrix as the rest. > For all these reasons the display analogy really is a bit fit for these > keyboards > we tried to come up with a universal coordinate system for these at the > beginning > of the thread and we failed ... I'd suggest trying harder this time :-). Best regards, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. signature.asc Description: PGP signature
Re: Future handling of complex RGB devices on Linux v2
Hi! > > > so after more feedback from the OpenRGB maintainers I came up with an even > > > more generic proposal: > > > https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 > > > > > >evaluate-set-command ioctl taking: > > > >{ > > > > enum command /* one of supported_commands */ > > > > union data > > > > { > > > > char raw[3072], > > > > { > > > > > > > > }, > > > > Yeah, so ... this is not a interface. This is a backdoor to pass > > arbitrary data. That's not going to fly. > > > > For keyboards, we don't need complete new interface; we reasonable > > extensions over existing display APIs -- keyboards are clearly 2D. > > I suppose they could be seen as *a* display, but if you are referring > to DRM KMS UAPI, then no, I don't see that fitting at all: So -- we already have very similar displays in drivers/auxdisplay. drivers/auxdisplay/cfag12864b.c is 128x64 display, 1-bit display for example. > - the "pixel grid" is not orthogonal, it's not a rectangle, and it > might not be a grid at all It is quite close to orthogonal. I'd suggest simply pretending it is orthogonal grid with some pixels missing :-). We already have cellphone displays with rounded corners and holes in them, so I suspect handling of missing pixels will be neccessary anyway. > - Timings and video modes? DRM KMS has always been somewhat awkward for > display devices that do not have an inherent scanout cycle and timings > totally depend on the amount of pixels updated at a time > (FB_DAMAGE_CLIPS), e.g. USB displays (not USB-C DP alt mode). > They do work, but they are very different from the usual hardware > involved with KMS, require special consideration in userspace, and > they still are actual displays while what we're talking about here > are not. As you say, there are other displays with similar problems. > - KMS has no concept of programmed autonomous animations, and likely > never will. They are not useful with actual displays. Yep. We need some kind of extension here, and this is likely be quite vendor-specific, as animations will differ between vendors. I guess "please play pattern xyzzy with parametrs 3 and 5" might be enough for this. > - Userspace will try to light up KMS outputs automatically and extend > the traditional desktop there. This was already a problem for > head-mounted displays (HMD) where it made no sense. That was worked > around with an in-kernel list of HMDs and some KMS property > quirking. This will need fixing for cfag12864b.c, no? Perhaps userspace should simply ignore anything smaller than 240x160 or something... > Modern KMS UAPI very much aims to be a generic UAPI that abstracts > display devices. It already breaks down a little for things like USB > displays and virtual machines (e.g. qemu, vmware, especially with > remote viewers), which I find unfortunate. With HMDs the genericity > breaks down in other ways, but I'd claim HMDs are a better fit still > than full-featured VM virtual displays (cursor plane hijacking). With > non-displays like keyboards the genericity would be completely lost, as > they won't work at all the same way as displays. You cannot even show > proper images there, only coarse light patterns *IF* you actually know > the pixel layout. But the pixel layout is(?) hardware-specific which is > the opposite of generic. > > While you could dress keyboard lights etc. up with DRM KMS UAPI, the > userspace would have to be written from scratch for them, and you > somehow need to make existing KMS userspace to never touch those > devices. What's the point of using DRM KMS UAPI in the first place, > then? Well, at least we have good UAPI to work with. Other options were 100 files in /sys/class/leds, pretending it is a linear array of leds, just passing raw data around, and pretending it is grid of RGB888 data. Anyway, there are devices such as these: (8x8 LED display). https://www.laskakit.cz/8x8-led-matice-s-max7219-3mm-cervena/ We should think about what interface we want for these, and then I believe we should make RGB keyboards use something similar. Best regards, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. signature.asc Description: PGP signature
Re: Future handling of complex RGB devices on Linux v2
Hi! > > Yeah, so ... this is not a interface. This is a backdoor to pass > > arbitrary data. That's not going to fly. > > Pavel, Note the data will be interpreted by a kernel driver and > not passed directly to the hw. Yes, still not flying :-). > With that said I tend to agree that this seems to be a bit too > generic. Exactly. > Given that these devices are all different in various ways > and that we only want this for devices which cannot be accessed > from userspace directly (so a limit set of devices) I really > think that just doing custom ioctls per vendor is best. I don't think that's good idea in the long term. Kernel should provide hardware abstraction, so that userspace does not need to know about hardware. Obviously there are exceptions, but this should not be one of those. BR, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. signature.asc Description: PGP signature
Re: Future handling of complex RGB devices on Linux v2
This certainly is the most KISS approach. This proposal in essence is just an arbitrary command multiplexer / demultiplexer and ioctls already are exactly that. With the added advantage of being able to directly use pass the vendor-cmd-specific struct to the ioctl instead of having to first embed it in some other struct. There's also the question of how much complexity needs to remain in the kernel, if vendor-specific ioctls are made available. Does every vendor driver implement a complex mapping to hardware registers? What about drivers that basically implement no mapping at all and simply forward all data to the hardware without any checking? The latter case would match Pavel's concerns, although I don't see how this is any different from the situation today, where userspace talks directly to the hardware via libusb etc. To be honest, I think the kernel shouldn't include too much high-level complexity. If there is a desire to implement a generic display device on top of the RGB device, this should be a configurable service running in user space. The kernel should provide an interface to expose this emulated display as a "real" display to applications - unless this can also be done entirely in user space in a generic way.
Re: Future handling of complex RGB devices on Linux v2
Hi, On 2/22/24 12:38, Gregor Riepl wrote: >> This certainly is the most KISS approach. This proposal >> in essence is just an arbitrary command multiplexer / >> demultiplexer and ioctls already are exactly that. >> >> With the added advantage of being able to directly use >> pass the vendor-cmd-specific struct to the ioctl instead >> of having to first embed it in some other struct. > > There's also the question of how much complexity needs to remain in the > kernel, if vendor-specific ioctls are made available. > > Does every vendor driver implement a complex mapping to hardware registers? > What about drivers that basically implement no mapping at all and simply > forward all data to the hardware without any checking? The latter case would > match Pavel's concerns, although I don't see how this is any different from > the situation today, where userspace talks directly to the hardware via > libusb etc. This whole discussion got started by embedded-controller driven keyboards in laptops with per key RGB lighting. We cannot just allow userspace raw-access to the embedded-controller. So these per vendor ioctl commands will need to do the minimum to make sure userspace cannot do bad things. But yes complex stuff like figuring out which LED(s) maps to say the enter key should be left to userspace. Especially since this can differ per keyboardlayout. > To be honest, I think the kernel shouldn't include too much high-level > complexity. If there is a desire to implement a generic display device on top > of the RGB device, this should be a configurable service running in user > space. The kernel should provide an interface to expose this emulated display > as a "real" display to applications - unless this can also be done entirely > in user space in a generic way. We really need to stop seeing per key addressable RGB keyboards as displays: 1. Some "pixels" are non square 2. Not all "pixels" have the same width-height ratio 3. Not all rows have the same amount of pixels 4. There are holes in the rows like between the enter key and then numpad 5. Some "pixels" have multiple LEDs beneath them. These might be addressable per LEDs are the sub-pixels ? What about a 2 key wide backspace key vs the 1 key wide + another key (some non US layouts) in place of the backspace? This will be "2 pixels" in some layout and 1 pixel with maybe / maybe-not 2 subpixels where the sub-pixels may/may not be individually addressable ? For all these reasons the display analogy really is a bit fit for these keyboards we tried to come up with a universal coordinate system for these at the beginning of the thread and we failed ... Regards, Hans
Re: Future handling of complex RGB devices on Linux v2
Hi, On 2/21/24 23:17, Pavel Machek wrote: > Hi! > >> so after more feedback from the OpenRGB maintainers I came up with an even >> more generic proposal: >> https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 > >>> evaluate-set-command ioctl taking: >>> { >>> enum command /* one of supported_commands */ >>> union data >>> { >>> char raw[3072], >>> { >>> >>> }, > > Yeah, so ... this is not a interface. This is a backdoor to pass > arbitrary data. That's not going to fly. Pavel, Note the data will be interpreted by a kernel driver and not passed directly to the hw. With that said I tend to agree that this seems to be a bit too generic. Werner, it seems you are basically re-inventing ioctls here. If you are going to use per vendor specific data structs for various commands and have those defined in the kernel userspace API headers, then this means that userspace will already need updated versions of those headers to support new vendors / new laptop models if the commands change for a new model. So what you are basically providing here is a generic interface to pass a cmd number + a cmd-bumber-specific data struct and we already have that it is called an ioctl. So I think that the conclusion of this whole discussion is that with the exception of a get-dev-info ioctl, we simply want vendor specific ioctls, using 1 ioctl per command. Given that these devices are all different in various ways and that we only want this for devices which cannot be accessed from userspace directly (so a limit set of devices) I really think that just doing custom ioctls per vendor is best. This certainly is the most KISS approach. This proposal in essence is just an arbitrary command multiplexer / demultiplexer and ioctls already are exactly that. With the added advantage of being able to directly use pass the vendor-cmd-specific struct to the ioctl instead of having to first embed it in some other struct. Regards, Hans
Re: Future handling of complex RGB devices on Linux v2
On Wed, 21 Feb 2024 23:17:52 +0100 Pavel Machek wrote: > Hi! > > > so after more feedback from the OpenRGB maintainers I came up with an even > > more generic proposal: > > https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 > > > >evaluate-set-command ioctl taking: > > >{ > > > enum command /* one of supported_commands */ > > > union data > > > { > > > char raw[3072], > > > { > > > > > > }, > > Yeah, so ... this is not a interface. This is a backdoor to pass > arbitrary data. That's not going to fly. > > For keyboards, we don't need complete new interface; we reasonable > extensions over existing display APIs -- keyboards are clearly 2D. I suppose they could be seen as *a* display, but if you are referring to DRM KMS UAPI, then no, I don't see that fitting at all: - the "pixel grid" is not orthogonal, it's not a rectangle, and it might not be a grid at all - Timings and video modes? DRM KMS has always been somewhat awkward for display devices that do not have an inherent scanout cycle and timings totally depend on the amount of pixels updated at a time (FB_DAMAGE_CLIPS), e.g. USB displays (not USB-C DP alt mode). They do work, but they are very different from the usual hardware involved with KMS, require special consideration in userspace, and they still are actual displays while what we're talking about here are not. - KMS has no concept of programmed autonomous animations, and likely never will. They are not useful with actual displays. - Userspace will try to light up KMS outputs automatically and extend the traditional desktop there. This was already a problem for head-mounted displays (HMD) where it made no sense. That was worked around with an in-kernel list of HMDs and some KMS property quirking. Modern KMS UAPI very much aims to be a generic UAPI that abstracts display devices. It already breaks down a little for things like USB displays and virtual machines (e.g. qemu, vmware, especially with remote viewers), which I find unfortunate. With HMDs the genericity breaks down in other ways, but I'd claim HMDs are a better fit still than full-featured VM virtual displays (cursor plane hijacking). With non-displays like keyboards the genericity would be completely lost, as they won't work at all the same way as displays. You cannot even show proper images there, only coarse light patterns *IF* you actually know the pixel layout. But the pixel layout is(?) hardware-specific which is the opposite of generic. While you could dress keyboard lights etc. up with DRM KMS UAPI, the userspace would have to be written from scratch for them, and you somehow need to make existing KMS userspace to never touch those devices. What's the point of using DRM KMS UAPI in the first place, then? Thanks, pq pgp0rYnjXBjKY.pgp Description: OpenPGP digital signature
Re: Future handling of complex RGB devices on Linux v2
Hi! > so after more feedback from the OpenRGB maintainers I came up with an even > more generic proposal: > https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3916#note_1753072869 > >evaluate-set-command ioctl taking: > >{ > > enum command /* one of supported_commands */ > > union data > > { > > char raw[3072], > > { > > > > }, Yeah, so ... this is not a interface. This is a backdoor to pass arbitrary data. That's not going to fly. For keyboards, we don't need complete new interface; we reasonable extensions over existing display APIs -- keyboards are clearly 2D. Best regards, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. signature.asc Description: PGP signature