Re: nxgl Redrawing
If it is not too device specific, I can put it on a branch and we can haggle through the details before merging it to master. I think your original proposal for nx_updatedisplay() would be okay if it were renamed and took three arguments like: int nx_synch(NXWINDOW hwnd, int cmd, unsigned long arg) it would be prototyped in include/nuttx/nx/nx.h and there would probably need to be a new include/nuttx/nx/nx_ioctl.h to hold IOCTL commands and structures. These are things I can help with. Greg Yes, I already did some name changes because they were too close to already used terms. Let me build this strawman and then we've got something to work from. Regards DAVE
Re: nxgl Redrawing
On 12/17/2019 10:10 AM, Gregory Nutt wrote: So, I think the path here is that I'll tidy up what I've done and then I'll submit it for comment and kickabout, and see where we go from there. If it is not too device specific, I can put it on a branch and we can haggle through the details before merging it to master. I think your original proposal for nx_updatedisplay() would be okay if it were renamed and took three arguments like: int nx_synch(NXWINDOW hwnd, int cmd, unsigned long arg) it would be prototyped in include/nuttx/nx/nx.h and there would probably need to be a new include/nuttx/nx/nx_ioctl.h to hold IOCTL commands and structures. These are things I can help with. Greg
Re: nxgl Redrawing
So, I think the path here is that I'll tidy up what I've done and then I'll submit it for comment and kickabout, and see where we go from there. If it is not too device specific, I can put it on a branch and we can haggle through the details before merging it to master.
Re: nxgl Redrawing
Linux uses Xorg for graphics. So if this feature is supported in Linux, then I think it would be through Xorg? The NX server is the tiny, moral equivalent of the XServer. This is interesting: https://hackaday.com/2018/08/14/run-a-linux-terminal-on-cheap-e-ink-displays/ The interface is written in Python: https://github.com/joukos/PaperTTY but I don't really see anything of interest. I see a lot of references to "deferred IO" with ePaper devices. AFAIK deferred IO is basically what drivers/lcd/lcd_framebuffer.c does: If buffers all writes to the device in a framebuffer. You could then force the framebuffer dump to hardware using the FBIO_UPDATE ioctl in drivers/video/fb.c Sorry... a half hour of Googling did not answer the question for me. I'm not aware of any 'standard' interfaces. Most of the code I've seen is hackup bits and pieces, but there's no way I'm going to build a whole parallel ecosystem when we've got all the juicy stuff to hand...provided we don't break anything to use it! For interest, you can see the sort of modules I'm using at https://www.buydisplay.com/default/e-paper-display-module-e-ink-display-kit-manufacturers/2-7-inch So, I think the path here is that I'll tidy up what I've done and then I'll submit it for comment and kickabout, and see where we go from there. Regards DAVE
Re: nxgl Redrawing
On 12/17/2019 9:03 AM, Dave Marples wrote: How are other operating systems / graphics libraries handling this? From what I've seen they mostly don't...you end up with a parallel graphics subsystem built on top of an SPI. We can do better than that, given that very little needs to be extended. Linux uses Xorg for graphics. So if this feature is supported in Linux, then I think it would be through Xorg? The NX server is the tiny, moral equivalent of the XServer. This is interesting: https://hackaday.com/2018/08/14/run-a-linux-terminal-on-cheap-e-ink-displays/ The interface is written in Python: https://github.com/joukos/PaperTTY but I don't really see anything of interest. I see a lot of references to "deferred IO" with ePaper devices. AFAIK deferred IO is basically what drivers/lcd/lcd_framebuffer.c does: If buffers all writes to the device in a framebuffer. You could then force the framebuffer dump to hardware using the FBIO_UPDATE ioctl in drivers/video/fb.c Sorry... a half hour of Googling did not answer the question for me.
Re: nxgl Redrawing
How are other operating systems / graphics libraries handling this? From what I've seen they mostly don't...you end up with a parallel graphics subsystem built on top of an SPI. We can do better than that, given that very little needs to be extended. Regards DAVE
Re: nxgl Redrawing
It means that some practical details of the implementation of the display technology are 'leaking' up into the graphics abstraction, but I don't really see a way to avoid it. Only the application knows when it's permissible to take an extended time to perform an update. How are other operating systems / graphics libraries handling this? I can imagine how minimal RTOS and bare metal solutions would do this -- via calls into hodge podge HAL. But that would not apply to a POSIX OS which constrains application interfaces to only portable interfaces. If Linux/Android supports this device, how they did this would be meaningful. If you provide the display info, I can check of there is support by other higher-level OSs. I generally accept how-linux-does-it as good as a specification in lack of any real specification. [Although I really don't like looking at Linux drivers these days. The Linux driver sub-subsystem has gotten very complex and I stopped keeping up with Linux internals years ago]. Greg
Re: nxgl Redrawing
On 12/17/19, Gregory Nutt wrote: > Thanks. I understand better now. > > This is very specific to this hardware.I don't think this should > involve the graphics system in such a device-specific way since this is > a hardware interfacing issue that has nothing to with graphics other > than a graphics device is involved. There are easy hacks or complex > architectural changes to accomplish what you want to do. > > For easy hacks, I would suggest a minimal, trivial device driver that > just supports the the FBIO_UPDATE command (and any other command unique > to the hardware). This driver would have to lie in the board-specific > logic or else have an SPI interface passed to it during initialization. > Or maybe a boardctl(BOARDIOC_IOCTL) command. > > If you wanted to support a graphics driver interface via NX, that is > okay too. But we would have to do that right, that is, in a > device-independent, platform-independent, general way. > > A better functional partitioning would be to have this hardware-specific > functionality implemented inside of the LCD driver, rather than in some > hack character driver or BOARDIOC_IOCTL. There is, however, no LCD > driver interface exposed to the application; the LCD driver interface is > only available to NX. I suppose it would be possible to add to an > ioctl() method to every LCD (or FB) driver and and nx_ioctl() method to > communicate and device-specific commands with the driver. But that > would be a pretty big effort (but a worthwhile contribution). I would > help on that one; the other, simpler ideas are basically hacks that > could give you the functionality you need. A generic graphics hardware > IOCTL command would besystematic and general and common to all graphics > devices. > Let's to follow the usual steps: how does Linux kernel implement the e-Ink/ePaper driver? Amazon developed the Kindle (their Lab126) more than 10 years ago and it uses Linux since its inception. So I think there is a well defined way to do it. Well, I didn't find much information about it, but only an old instruction to this subject: https://www.networkworld.com/article/2289160/kernel-space--e-paper-support-for-linux.html BR, Alan
Re: nxgl Redrawing
For easy hacks, I would suggest a minimal, trivial device driver that just supports the the FBIO_UPDATE command (and any other command unique to the hardware). This driver would have to lie in the board-specific logic or else have an SPI interface passed to it during initialization. Or maybe a boardctl(BOARDIOC_IOCTL) command. Thinking a little more: There is a serialization problem with using these easy hacks. The communication with NX is asynchronous through a message queue (this supports mult-threaded graphics clients). In order to synchronize with the NX server, you would need to do nx_synch() which blocks until the SYNC message is processed. l Then you can safely do the low level interface. If you wanted to support a graphics driver interface via NX, that is okay too. But we would have to do that right, that is, in a device-independent, platform-independent, general way. A better functional partitioning would be to have this hardware-specific functionality implemented inside of the LCD driver, rather than in some hack character driver or BOARDIOC_IOCTL. There is, however, no LCD driver interface exposed to the application; the LCD driver interface is only available to NX. I suppose it would be possible to add to an ioctl() method to every LCD (or FB) driver and and nx_ioctl() method to communicate and device-specific commands with the driver. But that would be a pretty big effort (but a worthwhile contribution). I would help on that one; the other, simpler ideas are basically hacks that could give you the functionality you need. A generic graphics hardware IOCTL command would besystematic and general and common to all graphics devices. An implementation of nx_ioctl() would be queued in order in the message queue and would not need any special synchronization. Greg
Re: nxgl Redrawing
Thanks. I understand better now. This is very specific to this hardware. I don't think this should involve the graphics system in such a device-specific way since this is a hardware interfacing issue that has nothing to with graphics other than a graphics device is involved. There are easy hacks or complex architectural changes to accomplish what you want to do. For easy hacks, I would suggest a minimal, trivial device driver that just supports the the FBIO_UPDATE command (and any other command unique to the hardware). This driver would have to lie in the board-specific logic or else have an SPI interface passed to it during initialization. Or maybe a boardctl(BOARDIOC_IOCTL) command. If you wanted to support a graphics driver interface via NX, that is okay too. But we would have to do that right, that is, in a device-independent, platform-independent, general way. A better functional partitioning would be to have this hardware-specific functionality implemented inside of the LCD driver, rather than in some hack character driver or BOARDIOC_IOCTL. There is, however, no LCD driver interface exposed to the application; the LCD driver interface is only available to NX. I suppose it would be possible to add to an ioctl() method to every LCD (or FB) driver and and nx_ioctl() method to communicate and device-specific commands with the driver. But that would be a pretty big effort (but a worthwhile contribution). I would help on that one; the other, simpler ideas are basically hacks that could give you the functionality you need. A generic graphics hardware IOCTL command would besystematic and general and common to all graphics devices. Greg On 12/17/2019 7:44 AM, Dave Marples wrote: Hi Greg, Thanks for the replies. Comments embedded; What is the interface to the ePaper device? Is it through a serial interface? Or a framebuffer interface? The interface is over SPI. It 'looks like' a normal lcd supported by lcd_dev_s. The particular one I have is write-only, but read/write ones exist too. What do you might by NXGL? It is library of simply drawing helpers. I take it you are not using NX, the full graphics system. It would update automatically and we would not be having this discussion. It's worth explaining a little bit more about how these devices work, then it'll be clearer, I hope. They have a display RAM just like an SPI or I2C interfaced LCD does. You write to it in the same way using similar commands. The difference is that the display RAM _does _not_ automatically get transferred onto the screen. That has to be done by a separate operation which, depending on the device in question, can take up to 15 seconds or so. NXGL offers a nice set of drawing primitives which remain applicable to these displays. There is no difference to NXGL or the way that it is used, and no code in NXGL is changed to allow it to work with ePaper. Unfortunately, nothing changes on the screen until the send-to-screen (redraw) command is issued. Since you are redrawing something that was not drawn by NX, it must be something that you drew in your application? True? Are you using a software framebuffer? It was drawn by NX, it just didn't make it to the screen yet. The correct way to implement a software framebuffer driver for a serial graphics device is via drivers/lcd/lcd_framebuffer.c. That driver will allocate the software framebuffer and export a standard NuttX framebuffer driver interface. The framebuffer driver at drivers/video/fb.c will expose the framebuffer driver to the application. From the framebuffer driver you can use the FIOC_MMAP IOCTL command to get the address of the framebuffer. You would need enable CONFIG_LCD_UPDATE, then you can use FBIO_UPDATE command to force the software software framebuffer to be written to the serial hardware. A framebuffer isn't needed, as there is already one on the device. However, what I need to do is directly equivalent to FBIO_UPDATE. Normally FBIO_UPDATE works from NuttX RAM to the LCD (and automatically onto the display), I want to go from the ePaper RAM to the display. Using the framebuffer wouldn't help me as I'd still need to perform the redraw operation to get the stuff onto the actual display after sending the FBIO_UPDATE. Now, that sounds like you are using NX. NX will automatically update the display. I cannot imagine a situation where that would be necessary. Let's talk more before you waste your time. I would have to understand what you are really doing to be able to help you. NX _thinks_ it's updated the display and indeed, in the normal scheme of things (with an LCD), it would have. Unfortunately, for this class of displays, this second step to sync the display ram to the display itself is needed. I've just read through the other notes you've sent. To avoid fragmenting my reply; *) It's an ePaper electrophoretic display which is bi-stable -
Re: nxgl Redrawing
On Tue, Dec 17, 2019 at 8:44 AM Dave Marples wrote: > It means that some practical > details of the implementation of the display technology are 'leaking' up > into the graphics abstraction, but I don't really see a way to avoid it. > Only the application knows when it's permissible to take an extended > time to perform an update. How are other operating systems / graphics libraries handling this?
Re: nxgl Redrawing
Hi Greg, Thanks for the replies. Comments embedded; What is the interface to the ePaper device? Is it through a serial interface? Or a framebuffer interface? The interface is over SPI. It 'looks like' a normal lcd supported by lcd_dev_s. The particular one I have is write-only, but read/write ones exist too. What do you might by NXGL? It is library of simply drawing helpers. I take it you are not using NX, the full graphics system. It would update automatically and we would not be having this discussion. It's worth explaining a little bit more about how these devices work, then it'll be clearer, I hope. They have a display RAM just like an SPI or I2C interfaced LCD does. You write to it in the same way using similar commands. The difference is that the display RAM _does _not_ automatically get transferred onto the screen. That has to be done by a separate operation which, depending on the device in question, can take up to 15 seconds or so. NXGL offers a nice set of drawing primitives which remain applicable to these displays. There is no difference to NXGL or the way that it is used, and no code in NXGL is changed to allow it to work with ePaper. Unfortunately, nothing changes on the screen until the send-to-screen (redraw) command is issued. Since you are redrawing something that was not drawn by NX, it must be something that you drew in your application? True? Are you using a software framebuffer? It was drawn by NX, it just didn't make it to the screen yet. The correct way to implement a software framebuffer driver for a serial graphics device is via drivers/lcd/lcd_framebuffer.c. That driver will allocate the software framebuffer and export a standard NuttX framebuffer driver interface. The framebuffer driver at drivers/video/fb.c will expose the framebuffer driver to the application. From the framebuffer driver you can use the FIOC_MMAP IOCTL command to get the address of the framebuffer. You would need enable CONFIG_LCD_UPDATE, then you can use FBIO_UPDATE command to force the software software framebuffer to be written to the serial hardware. A framebuffer isn't needed, as there is already one on the device. However, what I need to do is directly equivalent to FBIO_UPDATE. Normally FBIO_UPDATE works from NuttX RAM to the LCD (and automatically onto the display), I want to go from the ePaper RAM to the display. Using the framebuffer wouldn't help me as I'd still need to perform the redraw operation to get the stuff onto the actual display after sending the FBIO_UPDATE. Now, that sounds like you are using NX. NX will automatically update the display. I cannot imagine a situation where that would be necessary. Let's talk more before you waste your time. I would have to understand what you are really doing to be able to help you. NX _thinks_ it's updated the display and indeed, in the normal scheme of things (with an LCD), it would have. Unfortunately, for this class of displays, this second step to sync the display ram to the display itself is needed. I've just read through the other notes you've sent. To avoid fragmenting my reply; *) It's an ePaper electrophoretic display which is bi-stable - whatever is on the display will stay there, even without power. That means transferring the contents of the display frame memory to the display is a second step operation which takes a while to happen. *) I can use nxgl to write to it no problem. As far as nxgl is concerned it looks just like an ILI9341 or similar (different command set, but the same principles). The problem is that what is written to it does not end up on the display until a separate, time consuming, redraw command is issued. While the redraw is going on the display is away with the faries and will not communicate at all. That isn't a problem for us as new drawing commands will just back up in the NXGL MQ until it becomes available again. *) There is no modification needed to any existing nxgl or other code for this to work...it's a simple extension to add this redraw command...but I need to be comfortable that I've put it in the right place, hence why I'm asking for input. It means that some practical details of the implementation of the display technology are 'leaking' up into the graphics abstraction, but I don't really see a way to avoid it. Only the application knows when it's permissible to take an extended time to perform an update. Regards DAVE
Re: nxgl Redrawing
Hmm...going further, the only way I see to deal with this at the nxgl layer above the lcd structure itself is to add a nx_redraw with corresponding NX_EVENT_REDRAWING/NXEVENT_REDRAWN event callbacks, but that makes the nxgl layer device-aware...I don't see how it can't be really, these things are pretty different to traditional LCDs. For the case of the lcd nx_updatedisplay member not being present I would just return the NXEVENT_REDRAWN immediately so everything would remain back-compatible. And you don't want to modified in the graphics interface unless you are also will to modify and test all of the graphic applications in apps/graphics that use it. There won't be any special change to the graphic system to support your case. If you could explain what you think this nx_displayupdate() function would do, that would also be helpful. Greg
Re: nxgl Redrawing
I don't really see any way to avoid this. Greg, any suggestions for a better approach? I would needed to understand how you are forming these displays, where the display resides (in application memory), and how you are interfacing from the "display" to hardware. I really don't have a clue what you are doing. We are not communicating.
Re: nxgl Redrawing
... I want to go the other way - when the application has decided that a new display is 'stable', it requests for it to be put on the screen. I don't understand what that means. Where is this new display?
Re: nxgl Redrawing
I need to understand more before I could comment. I've implemented an ePaper driver under nxgl, but as some of you are probably aware there needs to be an explicit redraw request to update an ePaper display. What is the interface to the ePaper device? Is it through a serial interface? Or a framebuffer interface? I'm not quite sure how to do that under nxgl as normally it would be driven automatically. It also takes a fair bit of time (~14 seconds for a colour display) during which period access to the ePaper is 'locked out' so the application really does need to be aware of what is going on. What do you might by NXGL? It is library of simply drawing helpers. I take it you are not using NX, the full graphics system. It would update automatically and we would not be having this discussion. Since you are redrawing something that was not drawn by NX, it must be something that you drew in your application? True? Are you using a software framebuffer? The correct way to implement a software framebuffer driver for a serial graphics device is via drivers/lcd/lcd_framebuffer.c. That driver will allocate the software framebuffer and export a standard NuttX framebuffer driver interface. The framebuffer driver at drivers/video/fb.c will expose the framebuffer driver to the application. From the framebuffer driver you can use the FIOC_MMAP IOCTL command to get the address of the framebuffer. You would need enable CONFIG_LCD_UPDATE, then you can use FBIO_UPDATE command to force the software software framebuffer to be written to the serial hardware. Here are some examples: $ find boards -name defconfig | xargs grep CONFIG_LCD_FRAMEBUFFER boards/arm/stm32/stm3240g-eval/configs/fb/defconfig:CONFIG_LCD_FRAMEBUFFER=y boards/arm/stm32/stm32f4discovery/configs/max7219/defconfig:CONFIG_LCD_FRAMEBUFFER=y boards/arm/stm32l4/nucleo-l476rg/configs/nxdemo/defconfig:CONFIG_LCD_FRAMEBUFFER=y My intention is to add an nx_updatedisplay() API...but is there a better, or already available, way to do this? It could be done via a call to nx_setvisibility on the background window I guess, but personally I don't like overlaid semantics like that... Now, that sounds like you are using NX. NX will automatically update the display. I cannot imagine a situation where that would be necessary. Let's talk more before you waste your time. I would have to understand what you are really doing to be able to help you. Greg
Re: nxgl Redrawing
Hmm...going further, the only way I see to deal with this at the nxgl layer above the lcd structure itself is to add a nx_redraw with corresponding NX_EVENT_REDRAWING/NXEVENT_REDRAWN event callbacks, but that makes the nxgl layer device-aware...I don't see how it can't be really, these things are pretty different to traditional LCDs. For the case of the lcd nx_updatedisplay member not being present I would just return the NXEVENT_REDRAWN immediately so everything would remain back-compatible. I don't really see any way to avoid this. Greg, any suggestions for a better approach? Regards DAVE On 17/12/2019 10:56, Dave Marples wrote: Folks, I've implemented an ePaper driver under nxgl, but as some of you are probably aware there needs to be an explicit redraw request to update an ePaper display. I'm not quite sure how to do that under nxgl as normally it would be driven automatically. It also takes a fair bit of time (~14 seconds for a colour display) during which period access to the ePaper is 'locked out' so the application really does need to be aware of what is going on. My intention is to add an nx_updatedisplay() API...but is there a better, or already available, way to do this? It could be done via a call to nx_setvisibility on the background window I guess, but personally I don't like overlaid semantics like that... Regards DAVE
Re: nxgl Redrawing
Hi Alan,
I'm no expert on this stuff either but as I understand things the
nx_callback/redraw, is for when nx requests that the client redraws a
portion of the screen (See 2.3.4.1 of the NX Graphics Subsystem manual).
I want to go the other way - when the application has decided that a new
display is 'stable', it requests for it to be put on the screen.
I don't really see how to do that in a standard way yet, so I've added a
new method to lcd.c::struct lcd_dev_s called 'redraw', but I don't want
to be making extensions to APIs without there being some element of
discussion first...
Regards
DAVE
On 17/12/2019 11:34, Alan Carvalho de Assis wrote:
Hi Dave,
On 12/17/19, Dave Marples wrote:
Folks,
I've implemented an ePaper driver under nxgl, but as some of you are
probably aware there needs to be an explicit redraw request to update an
ePaper display.
Other guy already used ePaper with NX Graphic libs, if I'm not wrong
it was Petteri Aimonen.
He also used NuttX nxgl in his Master Thesis:
http://jpa.kapsi.fi/stuff/other/diplomityo.pdf
I'm not quite sure how to do that under nxgl as normally it would be
driven automatically. It also takes a fair bit of time (~14 seconds for
a colour display) during which period access to the ePaper is 'locked
out' so the application really does need to be aware of what is going on.
My intention is to add an nx_updatedisplay() API...but is there a
better, or already available, way to do this? It could be done via a
call to nx_setvisibility on the background window I guess, but
personally I don't like overlaid semantics like that...
I'm not expert on graphics, but I think you can use the redraw in the
nx_callback to do it.
Please take a look at apps/examples/nxlines, mainly on it:
const struct nx_callback_s g_nxlinescb =
{
nxlines_redraw, /* redraw */
...
BR,
Alan
Re: nxgl Redrawing
Hi Dave,
On 12/17/19, Dave Marples wrote:
> Folks,
>
> I've implemented an ePaper driver under nxgl, but as some of you are
> probably aware there needs to be an explicit redraw request to update an
> ePaper display.
>
Other guy already used ePaper with NX Graphic libs, if I'm not wrong
it was Petteri Aimonen.
He also used NuttX nxgl in his Master Thesis:
http://jpa.kapsi.fi/stuff/other/diplomityo.pdf
> I'm not quite sure how to do that under nxgl as normally it would be
> driven automatically. It also takes a fair bit of time (~14 seconds for
> a colour display) during which period access to the ePaper is 'locked
> out' so the application really does need to be aware of what is going on.
>
> My intention is to add an nx_updatedisplay() API...but is there a
> better, or already available, way to do this? It could be done via a
> call to nx_setvisibility on the background window I guess, but
> personally I don't like overlaid semantics like that...
>
I'm not expert on graphics, but I think you can use the redraw in the
nx_callback to do it.
Please take a look at apps/examples/nxlines, mainly on it:
const struct nx_callback_s g_nxlinescb =
{
nxlines_redraw, /* redraw */
...
BR,
Alan
