Hello! I'd like to use JavaFX for the UI of an application that will involve rendering using an existing Vulkan-based renderer. For the sake of example, assume that the application looks and behaves a bit like the Unreal Engine 4 editing tools. Here's an example of those:
https://www.youtube.com/watch?v=2UowdJetXwA My understanding right now is that there isn't direct support in JavaFX for building this kind of application, and the primary reason for this is that there's a sort of conceptual wrestling match for control of a platform-specific rendering context here. For example: * A JavaFX application will tell JavaFX to open a new window, and the JavaFX implementation will do all of the OS-windowing-system-specific things to achieve this, and will also set up a system-specific rendering context depending on what the underlying platform is (OpenGL, DirectX, Metal, etc). JavaFX then translates input such as mouse and keyboard events from OS-specific types to the platform-independent JavaFX types so that the application can process them. * A typical Vulkan application will ask something analogous to the GLFW library to open a new window, and set up a rendering context. The GLFW library then translates input such as mouse and keyboard events from OS-specific types to generic GLFW event types, and the Vulkan application (probably) translates these into its own application-specific event types for processing. Obviously, in a given application, we're limited to having either one of these things happen, but realistically not both. The current approach (as of JavaFX 14) seems to be to use the PixelBuffer API in order to provide a CPU-side bridge between JavaFX and whatever rendering system is being used for external 3D rendering. In other words, this is the expected setup: 1. A JavaFX application will tell JavaFX to open a new window, and JavaFX will do all of the system-specific work required as described previously. 2. The application will then tell a library such as GLFW to create an _offscreen_ rendering context, perhaps configuring Vulkan or OpenGL. 3. The application, at the end of each frame, copies the contents of the offscreen rendering context's framebuffer into a PixelBuffer instance to be displayed inside a JavaFX UI. This, as far as I know, works correctly. The main complaint with this is that it pays a pretty heavy price: There's one framebuffer-sized copy operation from the GPU to the CPU (in order to read the required pixels), and then another framebuffer-sized copy operation back from the CPU to the GPU (either when writing into the PixelBuffer, or when JavaFX renders the contents of that PixelBuffer to the screen). My understanding is that it's essentially necessary to do these two rather expensive copying operations merely because JavaFX can't and won't expose the underlying rendering context it uses for its own UI rendering, and it also can't be expected to talk to whatever other rendering system the application might be using. The problem is essentially "we have these two systems both using the GPU, but they don't know each other and therefore we can't write code to get memory from one to the other without going via the CPU". Is this an accurate picture of the situation? As someone working exclusively with Vulkan, I can arrange to have the GPU copy the framebuffer into host-visible (not necessarily host-resident, but host _visible_) memory at the end of each frame. It's a little sad to have to actually copy that memory over the PCI bus just to immediately copy it back again, though. Is there no design we could come up with that would allow for at worst a simple GPU → GPU copy? I'm resigned to the fact that a copying operation is probably going to happen _somewhere_, but it'd be nice if we could avoid a rather expensive and redundant GPU → CPU → GPU copy. -- Mark Raynsford | https://www.io7m.com
