On 2021-02-15T13:40:43 +0000
Mark Raynsford <org.open...@io7m.com> wrote:
> Hello!
>
> I'd like to use JavaFX for the UI of an application that will
> involve rendering using an existing Vulkan-based renderer. For the sake
> of example, assume that the application looks and behaves a bit like
> the Unreal Engine 4 editing tools....
Apologies for the delay in responding (I've read and digested every
reply).
So... Having read everything, I'm reasonably sure that the ability to
do this in the "traditional" way - that is, having a node in the
JavaFX scene graph that displays the contents of a GPU-side
image/texture - isn't going to be happening any time soon. There are
multiple intersecting issues that make this intractably difficult to
achieve:
1. There are software boundaries; JavaFX doesn't expose the internal
rendering context it uses and, even if it did, there is a
combinatorial explosion of possibilities with regards to which
rendering API the _user_ might be using, and which rendering API
JavaFX might be using (if any!). Traditional stateful APIs like
OpenGL make it far too easy for external code to screw up the
rendering context anyway, so exposing JavaFX's own context would
be a bad idea.
2. There are hardware and concurrency issues; Let's assume that
JavaFX is using OpenGL or DirectX internally. Let's also assume
that the user is using some kind of API that explicitly provides
for managing concurrency with primitives such as fences,
semaphores, etc. The user would have to somehow magically arrange
for an image transfer on the GPU into an image that JavaFX knew
about, and would have to somehow set up all of the right memory
barriers and notify JavaFX when the transfer was completed. This
is likely to be intensely error-prone at the very least. The
thought of implementing this directly brings me out in a cold
sweat.
So how about another approach entirely?
In the past, I've rendered UI elements using entirely software rendering
and then uploaded the software-rendered UI as a texture to be overlaid
on top of the 3D rendered view. I could obviously continue to do this,
but maintaining entire software-rendered UI toolkits myself is a burden,
and it would be great if I could use an existing known-good toolkit
instead like JavaFX.
JavaFX _does_ have a software renderer. What if we could have JavaFX
work with entirely software rendering into an offscreen BufferedImage?
To make this work, we'd need the following:
1. We'd need to be able to initialize JavaFX in a manner that
_didn't_ require JavaFX to "own" the primary stage. That is,
right now JavaFX will invoke a user's Application subclass
with a Stage that JavaFX created by itself. This won't work
for most rendering engines, as typically the programmer has
to do a fair bit of work to open a graphics context with
the correct settings. Consider the typical code needed to
open a window over GLFW with Vulkan...
https://github.com/io7m/jcoronado/blob/develop/com.io7m.jcoronado.examples/src/main/java/com/io7m/jcoronado/examples/HelloVulkan.java
2. We'd need to be able to force JavaFX to operate entirely
using Java2D software rendering. This shouldn't be a problem
at all.
3. We'd need to be able to make JavaFX render to an offscreen
BufferedImage. It would be the responsibility of the programmer
to submit keyboard and mouse events to JavaFX manually, because
JavaFX would no longer "own" the primary application window.
I believe the first half of this is already doable in the sense
that the Node class allows for taking snapshots of a scene; these
snapshots could be uploaded to the GPU as textures. The second
half I believe is not exposed in any sense; JavaFX opens its own
windows, and consumes input events from those windows directly.
I wonder if it would be possible to, for example, add some kind of
WindowFactory API to JavaFX so that, when JavaFX wants to create a
Window, there's the option to delegate window creation to the
user. The created windows might not even be operating system windows
(for window-in-window UI systems)[0]. As long as the returned Window
instance acted like a Window in terms of returning dimensions,
producing the right input events... Would JavaFX even know that it
wasn't dealing with an operating system window?
[0]
https://softcamel.com/wp-content/uploads/2019/03/Transport-Tycoon-Deluxe-2.png
--
Mark Raynsford | https://www.io7m.com
