On 2021-02-16T16:09:04 +0000 Neil C Smith <n...@codelerity.com> wrote: > > I agree with you, and have certain similar requirements, like being > able to allow GStreamer and JavaFX to share GPU contexts. In fact, > was bugging Johan about this in the chat around his FOSDEM talk, and > promised to follow up here, so might as well pop my head above the > parapet. :-)
I do think it ultimately boils down to "provide an image to JavaFX when it asks for one, and don't allow the image to leave the GPU". I've used GStreamer outside of Java quite a bit for aggregating feeds from network cameras... In your case, is it that you want to consume one or more video streams from GStreamer inside a JavaFX UI? We'd be pretty close in terms of requirements with regards to not copying across CPU/GPU boundaries, and requiring low latency. :) > I certainly don't know what such an API should look like, but in some > ways to me parallels the differences between io file and nio2 files - > hidden by abstraction vs type-safe queryable capabilities / profiles. > In fact, given above, also something slightly akin to GStreamer's > context querying. > > Incidentally, reading your initial post about PixelBuffer reminds me > that I should also follow up on a point about concurrency issues with > that from last year. The API has (or had?) issues with accessing the > buffer from the rendering thread after it's been removed in the event > thread, which particularly with externally allocated buffers makes it > hard to safely mark as invalid to allow them to be freed. Yeah, that is a problem. It's why I said in my original email that I was resigned to the fact that a copy was probably going to have to happen somewhere. Without going into API specifics, I was sort of picturing JavaFX maintaining two images on the GPU; at any given time one is being displayed (ie. used as a texture whilst rendering a scene) and the other is being offered to the user as a destination for a GPU copy operation. It would be the responsibility of the user to copy an image in the right format into the offered buffer. There would need to be the appropriate memory barriers inserted, and I don't know how this could/would be handled given that it might be two completely different graphics APIs involved (there isn't a Vulkan JavaFX backend, for example,so in my case I'm guaranteed to be talking to something that isn't Vulkan). When an image had been copied in as required, and JavaFX had been told that this had happened, the two images would be swapped on the GPU. -- Mark Raynsford | https://www.io7m.com
