How are other operating systems / graphics libraries handling this?
From what I've seen they mostly don't...you end up with a parallel graphics subsystem built on top of an SPI. We can do better than that, given that very little needs to be extended.
Regards DAVE