Seems there are a few different issues here. I'll weigh in on a couple
of them. To summarize what Jim has already said:
* We always do GPU hardware rendering wherever possible
* Node caching was intended as a hint to the system to cache the result
of potentially expensive effects into a texture (which is rendered into
by the GPU) and re-render that texture when needed. Jim was suggesting
that an application might be able to use that mechanism to do something
other than its original purpose.
* Snapshot was intended to allow an application to take (as the name
suggests) a snapshot of a scene or node rendered into an image and
return that image to the application. We don't use snapshot internally,
rather various "render and then get the results" operations like
snapshot, node caching, and rendering to a JFXPanel all use similar
mechanisms which boil down to "render to texture" via OpenGL or D3D. We
have considered providing new API to allow more of a "live
render-to-texture" operation in the future, but haven't thought about it
recently.
Much of the discussion has been centered on what is possible with the
current JavaFX API, but there has also been some discussion on what
might be done with a new API.
Anyway, to comment on a couple specific points, I have some inline
comments below.
Jim Graham wrote:
On 8/13/2015 1:29 AM, Matthias Hänel wrote:
I'd argue that we sort of do have something like that - it is the
cache flag. If a Node is cached then we do have a copy of it in a
texture and that can help make the Blur Effect work more
efficiently, but there may be some additional copies between
textures if everything isn't set up right. Still, that is an
avenue for someone to check to see if there isn't a better way to
achieve this effect in the short term...
I am not pretty sure what cache does. Probably some hasmap that
holds objects and they are not instantly destroyed in the graphics
RAM?
It is not a hashmap.
It is a hint to save the rendering of that node in a buffer:
https://docs.oracle.com/javase/8/javafx/api/javafx/scene/Node.html#cacheProperty
I don't like the way that this doc comment is worded as it implies
that using it on a node that is blurred is unwise, but if the node
is animated over even GPU acceleration of the rendering and blurring
operations" are going to have some cost that it could save.
That documentation says nodes are cached as Bitmaps. In GPU or in CPU
space? It is not clear here. I suspect it is in CPU RAM space.
That will just lead to more copy-tasks from CPU to GPU or not? If
optimized there is no benefit at all as stated in the docu "note that
on some platforms such as GPU accelerated platforms there is little
benefit".
As I said, the doc comments there are confusing. "bitmap" here,
despite any capitalization, is a general term for "some type of pixel
store". On GPU accelerated platforms, this is a vram texture/FBO.
Actually, I would expect that renered Nodes are textures in GPU-VRAM
to get the most performance out of it.
Yes, nodes with the cache hint are rendered into VRAM textures and
reused for subsequent frames if the cache hint is set - subject to the
policies of the cacheHint property if any (mainly transform)
properties of the node are changed in the meantime.
From my current point the major problem with JavaFX is still the
same.
1. Has a good API
2. renders most of its stuff in software, hence does not run
performant
3. Has good approaches, but the overall sight on the technology is
broken somewhere.
I am not sure how you come to the conclusion that it renders most of
its stuff in software. It renders quite a lot in hardware. Even
the example here of using snapshot to optimize a blurred background
- the rendering of the scene is done in hw. It is only copied to
main memory because the API requires a persistent image. If you
render that image to the screen it is copied back into a texture and
reused from that texture unless we run low on vram. There is no
rendering in software there, only use of a heap buffer for
persistent storage...
Your point is that it uses hardware to render and it just uses
snapshot to satisfy the API?
What is "it"?
JavaFX doesn't "use" snapshot anywhere. It provides the snapshot API
so the developer can use snapshot where the developer wants to, but
JavaFX doesn't invoke the snapshot API on its own.
In particular, the node cache hint does not use the snapshot API under
the covers.
My current understaning is ... what happens in our blurred-effect case?
To be clear, you are describing what happens in a particular
implementation of the blurred-effect case. The implementation that
uses snapshot, which may be the popular technique in use at this time.
1. the application constructs a JavaFX node tree.
2. the node tree is rendered mostly in hardware (shader effects and
so on on top) to the main framebuffer
Correct
3. Snapshot calls ReadPixels (or whatever it is called on the
particular platform)
4. JavaFX encapsulates this "new" image with Object
Technically, snapshot does its own (hardware accelerated) rendering
into a separate GPU texture/FBO and does a read pixels on that
texture. It then encapsulates those pixels into an Image object.
This snapshot operation need only be done once if the underlying node
tree is static.
5. We draw the "new" image with effects to the OGL context with the
same node-tree API as we did before on top of the first node-tree.
Correct. Note that when you render that Image object its pixels are
cached in vram and that vram copy is reused from frame to frame. So,
if you reuse the one snapshot then there was only one trip from vram
to memory and back to vram on the first frame you did this and then
everything should be done in vram for subsequent frames reusing the
same snapshot.
That works, but there is too much CPU and memcpy involved for my
believe. Furthermore ReadPixels takes forever in
an OpenGL perspective.
If the underlying tree is static, though, that readpixels operation
only happens once at the start of the operation, but you are correct
that it would be much better if it didn't need to happen at all.
Since this is just one very simple effect, it is actually not good to
spend more than approx. 20% CPU (i7) load on it.
I expect 0% (not noticable) for this blurry effect.
That would be ideal. We may need new API to get there, but there are
also options to consider that may get us there in the short term. In
particular, cached nodes - which are already present in the API - may
get us closer to that goal.
The ideal implementation from my perspective would be:
1. the application constructs a JavaFX node tree.
2. the node tree is rendered mostly in hardware (shader effects and
so on on top) to a virtual framebuffer in the GPU space
3. The virtual framebuffer is drawn by a simple drawVert-call for the
background
4. The virtual framebuffer is drawn once again shaped and shaded
(blurry filter) by another drawVert-call
If you set the cache hint to true on the underlying tree then this may
be approximately what happens in our current implementation. The part
I would still need to investigate would be how well the effects
machinery (called Decora) can reuse the cached version of the nodes.
If it doesn't attempt to reuse the cached version of the nodes then it
might end up re-rendering the tree.
This would an interesting experiment, and would let us know how easily
applications could do what they need with the existing API. If there are
limitations or performance bottlenecks we can look into fixing them.
I have to correct my assumption that JavaFX renders most of its stuff
in software a bit. The wording "rendering" was not correct.
This feeling comes probaly from the massive use of Snapshot. I have
not understood the entire core of Prism until now.
In some implementation of snapshot it seems to do the rendering
entirely in software. That might only the fallback, but the
ReadPixels-Stuff is still valid.
To be clear, snapshot was mainly created for its namesake - to produce
static copies of the scene data to be saved as images for putting into
documentation. Alternate uses that were considered were for
generating thumbnails of scenes for an application that could
open/close various scene graph panes. One wouldn't use the Windows
"print screen" API in a performance intensive part of one's
application either. It is great that it can be used for this
particular result, but it was not designed to be performant in that
respect. Note that the primary snapshot API is the asynchronous
version with the callback to deliver the data. The non-asynchronous
version is mostly just a helper around that, but it stalls the
rendering pipeline to complete its work. None of that was designed
for insertion into a running animation technique.
Exactly.
Tiny-Offtopic: I already meantioned on this list that JFXPanel
(Swing) is way below it's expectations. The main reason there is also
the use
of the snapshot function instead of letting render JavaFX in it's own
heavy weight window. In our small test case a simple List
displayed via JFXPanel was not even to render more than 1 fps and it
slowed down the entire Swing-UI. Since I know jogle and it's canvas
implementation, would it be good to have a similar heavy weight JFX
canvas in Swing? This would let JFX use it's full hardware rendered
power.
It doesn't use snapshot per-se, but it does do something similar.
Sharing contexts and GPU resources with AWT/Java2D is on a wish list,
but architecturally we aren't there. Kevin would know more about our
challenges on that front than me...
I haven't looked recently at what it would take to share resources
between Java2D and AWT, but that seems separate from the question of
whether we could use a heavy-weight JFXCanvas as opposed to the current
lightweight JFXPanel. We haven't considered that since there are
limitations with heavyweight components. I am not aware of any
performance problems that are as serious as "simple List displayed via
JFXPanel was not even to render more than 1 fps". Do you have a test
case? I do note that we did have a bug on Mac retina displays that was
recently fixed which killed performance. Was that what you were running
your 1fps program on?
-- Kevin
...jim
