Re: Paint Phase Debugging / Performance

[Matthew Elliot]

Hey Johan, thanks for the detailed answer.

Perhaps a bit more context as well... from our experience it seems about
10-12% of the setups we hit fall back to Software because of various issues
or predominantly or 'sadly' virtualisation like Citrix so we have to be
sure in SW mode things run optimally. It isn't necessarily a bad thing to
optimise for the lowest common denominator if you want a fast application
everywhere.
Given this learning I'm now looking to how ensure we somehow 'box' what a
minimum performance looks like and ensure with every change we don't really
get away from that + communicate costs clearly with the UX team.
That way when our developers step on land mines we detect the missing limbs
the same day.

That's the direction I'm currently thinking about and of course I can't be
the only one (hence the post).

I don't like it when people say that "JavaFX is slow" because if everything
is done right, JavaFX can be extremely fast.
But I also realise that it is very easy to make things very slow in JavaFX,
and in case "performance" is slow, it is important to pinpoint the problem
as easy as possible. And that is often not trivial, so the easier it
becomes to detect problems, the more developers will like JavaFX.

>>> Agree that JavaFX can be fast and given we have to support thousands of
setups that we have no control over I'm quite content with how JavaFX is
performing in the wild.
It definitely has a few performance traps that aren't so easy notice from
the high level API, css, effects, etc.
And while blocking the JavaFX thread happens the most frequent it also is
simple to explain to any reasonable developer and easy to troubleshoot as
it is something you influence directly, whereas css pass, layout or paint
are much more indirect and tricky.
An effect on top on an animation for example was very tricky to find. In
this case the framework should even perhaps refuse to do it in software
mode... :)

To a few other points:

The options in PrismSettings.java (e.g. prism.trace,
prism.printrendergraph, prism.showdirty) and in QuantumToolit
(quantum.debug) can be helpful as well.

>>> Are you aware of any write up on the internet about understanding the
output? I mean I had to read the code and use breakpoints for the prism
logger for 'slow background path for null' to understand but it would be
nice if someone has already written about these topics? There is also no
book on JavaFX perf right?
On the other side and regarding analysing issues I think there is also a
wealth of tooling / logging that is useful but it took me a while to work
my way into the performance investigation as the information is scattered
and or incomplete (at least with my googlefu).

It would be great to have tools that auto-detect this. Detecting slow
render phases is already done, but linking to the root cause is of course
much harder.
I don't think that interrupting the paint phase is a good thing. If that
takes 200-300 ms, it is very likely it will take 200-300ms in the next
cycle.

>>> Auto-detection is a hard science, hence why as a developer I wouldn't
mind setting explicitly for my CI perf builds, break if condition A happens
so I can review what changed recently.
>>> At 200-300 ms the application became unusable essentially in our case
as every scheduled pulse was either waiting or painting. I will create
something to also detect this going forward to avoid a repeat (question
just is what).
>>> Of course now I will redirect stdErr and parse it on the fly and/or go
in via reflection but it would be nice if prism would allow to set short
cuts or at least spit additional ERROR or WARNs.

It might be an option though to dynamically throttle the pulse frequency
(which can now only be changed at startup with javafx.animation.pulse) in
case slow rendering is detected.
>>> Does this matter much as the pulses anyway just get skipped, or? In
this case the fact that the animation is still trying to run when the pulse
is almost stalled doesn't make sense and circuit-breaking it would be
perhaps 'intelligent' framework behaviour.

When things are slow in the paint phase, the information about how many
Nodes are visited and rendered, and how many are cached is very useful. I
have a local debug version where I keep track of how many times a Node is
rendered from cache versus how often the cache is invalid and needs to be
recomputed (which is extremely expensive). It this was somehow automated,
it could improve performance. It is similar to what hotspot C1/C2 is doing
for code: it is expensive to compile all code at runtime, but when it turns
out methods are often used, they are compiled. Similar, if the rendering
pipeline detects that a node would remain cache-valid for most of the time,
it might automatically set that node.setCache(true). But that's not a
trivial thing to implement.
>>> I still can't 100% work out what exactly one should cache from what
I've googled. I mean, I only need to cache it if it will be redrawn right,

Re: Paint Phase Debugging / Performance

[Johan Vos]

Hi Matthew,

I agree this is a very important question.

I don't like it when people say that "JavaFX is slow" because if everything
is done right, JavaFX can be extremely fast.
But I also realise that it is very easy to make things very slow in JavaFX,
and in case "performance" is slow, it is important to pinpoint the problem
as easy as possible. And that is often not trivial, so the easier it
becomes to detect problems, the more developers will like JavaFX.

I typically start with setting prism.verbose and javafx.pulseLogger to
true, as you first need to know in which phase the problem is (e.g. layout
or render phase). If you see the problem is in the render phase, you have
to look at completely different things.
Note that in most cases I've seen with bad performance, the problem was not
in the rendering, but simply due to too much happening on the JavaFX
application thread, preventing it to start doing the rendering.
Those things are typically easily detected by generating thread dumps or
using simple profilers to find out what the JavaFX thread is typically
doing. If you see bad performance but in 90% of the thread dump there is no
trace of rendering, you know where to look.

When things are slow in the paint phase, the information about how many
Nodes are visited and rendered, and how many are cached is very useful. I
have a local debug version where I keep track of how many times a Node is
rendered from cache versus how often the cache is invalid and needs to be
recomputed (which is extremely expensive). It this was somehow automated,
it could improve performance. It is similar to what hotspot C1/C2 is doing
for code: it is expensive to compile all code at runtime, but when it turns
out methods are often used, they are compiled. Similar, if the rendering
pipeline detects that a node would remain cache-valid for most of the time,
it might automatically set that node.setCache(true). But that's not a
trivial thing to implement.

The options in PrismSettings.java (e.g. prism.trace,
prism.printrendergraph, prism.showdirty) and in QuantumToolit
(quantum.debug) can be helpful as well.

When there are no clear indications (e.g. not too many nodes, no invalid
caches), I go for profiling, working bottom-up. I have the JavaFX source
code always at hand when doing this, in order to see what exactly is
happening.
There are some patterns, e.g. on Android I know that lots of time spent in
System.arraycopy is an indication about lots of slow CSS processing (Bitset
operations, if you follow the profiling information bottom-up)

It would be great to have tools that auto-detect this. Detecting slow
render phases is already done, but linking to the root cause is of course
much harder.

I don't think that interrupting the paint phase is a good thing. If that
takes 200-300 ms, it is very likely it will take 200-300ms in the next
cycle.

It might be an option though to dynamically throttle the pulse frequency
(which can now only be changed at startup with javafx.animation.pulse) in
case slow rendering is detected.

- Johan


On Tue, May 1, 2018 at 8:17 PM Matthew Elliot <
matthew.james.ell...@gmail.com> wrote:

> Hi all,
>
> The last few days I was troubleshooting a new performance issue that showed
> up in our PROD application where customers had fallen back to the SW
> rendering pipeline. It severely affected the application where CPU
> frequency was under 3 GHz with hover lags of a few seconds in the worst
> cases. With thousands of potential HW/SW combinations in the wild it took
> quite a while to even identify it really was an issue in our application
> and not the usual noise of some silly set up. All this got me thinking...
>
> ... what was visible was long paint passes, and long waiting on previous
> render but narrowing this down to exactly what was going on took a lot of
> manual inspection of the rendering pipeline code / debugging and somewhat
> by chance I stumbled over the -Dprism.disableEffects flag which after much
> more pain helped me narrow down the issue.
>
> The root cause turned out to be an -fx-effect (blend, inner shadow) on an
> animated node that was set from the code by an unknowing developer.
>
> While there are tools like mission controller for visualising the pulse and
> phases it can be difficult to identify for example what is going wrong
> inside of the painting phase and it is difficult to control that nothing
> bad happens when many developers can make changes to the code and reviews
> will never catch everything... I'm therefore thinking about ways to run
> rendering tests in continuous integration that would fail fast if the SW
> rendering pipeline would get overloaded.
> I had a look at PulseListener where I could see pulse times but I'd like to
> go more detailed and actually like the information tracked in the internals
> of PulseLogger (PulseData) without doing any nasty tricks.
>
> I thought maybe somebody has already thought about this problem before and
> maybe there is even 

Re: native libs in modules

[Kevin Rushforth]

[including Alan and Mandy]

This came up briefly in a thread sent to jigsaw-dev [1], but with no 
real conclusion. I agree that it would be good to get a recommendation 
from the Jigsaw team.


After thinking about it more, here are my preferences:

* From the developer's point of view, they shouldn't have to know or 
care that some of our modules have platform-specific native code and 
others do not. Ideally, all an app developer should need is to declare a 
dependency on the needed module(s). Something like this:


    'javafx:javafx.controls:11'
    'javafx:javafx.graphics:11'   (although controls will depend on 
graphics, so this should be optional)


* For maven, the pom file should take care of downloading the jar 
file(s) for the specific platform; not sure if that is possible for 
gradle dependencies, but it would be ideal.


* For each module, we can package up the natives in one of two ways:

A) create a separate jar file per platform with the native code and the 
modular jar both in the same jar file. something like:


javafx-graphics-win-x64.jar
javafx-graphics-linux-x64.jar
javafx-graphics-osx-x64.jar

B) create one platform-independent modular jar file with all classes for 
all platforms, and also a separate jar per platform just for the native 
libraries:


javafx-graphics.jar  (with classes for all platforms)
javafx-graphics-natives-win-x64.jar
javafx-graphics-natives-linux-x64.jar
javafx-graphics-natives-osx-x64.jar

For either option there will need to be code that unpacks the natives.

I prefer option A if feasible, as I think it simplifies the build, 
hopefully without unduly complicating anything else. If it isn't 
feasible, then option B might be a good fallback.


Comments?

-- Kevin

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2018-April/013762.html


On 5/1/2018 5:34 AM, Johan Vos wrote:

I'd love to hear a general recommendation from the jigsaw team as well.
Clearly, there are a number of solutions, and as a developer, I easily 
get confused if some frameworks do it with option A and others with 
option B.

So what is the preferred approach in general?

It seems (given the large size of the webkit native library) there is 
a growing consensus here for bundling native code inside 
platform-specific jars?


- Johan

On Tue, May 1, 2018 at 7:38 AM Phil Race > wrote:


You are describing the situation today and making it into a
contract. You need to be sure that it is policed and enforceable.
Meaning there need to be tests to ensure it stays that way and
that it is not an unreasonable constraint on changes in the platform.

Also what if anything do the jigsaw team recommend ?

-Phil.

> On Apr 30, 2018, at 4:07 PM, Kevin Rushforth
>
wrote:
>
> The native libraries are quite large -- especially jfxwebkit --
and it does seem better to have per-platform jar files, at least
for the native libraries. The following modules could be
platform-independent since they have no natives:
>
> javafx.base
> javafx.controls
> javafx.fxml
> javafx.swing
>
> We would just need to test that the set of modular jar files for
each platform are the same, and then pick one (Linux?) to use for
generating the platform-independent jar files.
>
> The following modules have native libraries:
>
> javafx.graphics (*)
> javafx.media (*)
> javafx.web
>
> (*) - also has some differences in the set of class files that
are included depending on the platform
>
> So per-platform versions of the above would be needed to
accommodate the different native libraries.
>
> If it would help, the .class files could be always included for
all platforms, but there would be some additional effort to make
this work. Also, it might be just as easy to have the class files
and the natives in one downloaded jar file per platform, since at
least the natives need to be platform-independent. I'm far from a
maven expert, though, so I don't really know for sure which path
is easier.
>
> -- Kevin
>
>
>> On 4/30/2018 9:44 AM, Paul Ray Russell wrote:
>>  >I'm not sure I understand the question about
platform-specific jar files,
>>
>> Last time I worked on native specifics (which was to package up
RXTX dlls
>> for different OSs / in 64/32 bit) The easiest solution for pure
Maven
>> builds seemed to be, to package DLLs inside a jar. We then used
a profile
>> to control which DLL's were grabbed in different cases. And
surely for this
>> specific case, it would be better to split the native specifics
into
>> separate jars per OS to avoid a huge cross-OS download?
>

Re: native libs in modules

[Johan Vos]

I'd love to hear a general recommendation from the jigsaw team as well.
Clearly, there are a number of solutions, and as a developer, I easily get
confused if some frameworks do it with option A and others with option B.
So what is the preferred approach in general?

It seems (given the large size of the webkit native library) there is a
growing consensus here for bundling native code inside platform-specific
jars?

- Johan

On Tue, May 1, 2018 at 7:38 AM Phil Race  wrote:

> You are describing the situation today and making it into a contract. You
> need to be sure that it is policed and enforceable.
> Meaning there need to be tests to ensure it stays that way and that it is
> not an unreasonable constraint on changes in the platform.
>
> Also what if anything do the jigsaw team recommend ?
>
> -Phil.
>
> > On Apr 30, 2018, at 4:07 PM, Kevin Rushforth 
> wrote:
> >
> > The native libraries are quite large -- especially jfxwebkit -- and it
> does seem better to have per-platform jar files, at least for the native
> libraries. The following modules could be platform-independent since they
> have no natives:
> >
> > javafx.base
> > javafx.controls
> > javafx.fxml
> > javafx.swing
> >
> > We would just need to test that the set of modular jar files for each
> platform are the same, and then pick one (Linux?) to use for generating the
> platform-independent jar files.
> >
> > The following modules have native libraries:
> >
> > javafx.graphics (*)
> > javafx.media (*)
> > javafx.web
> >
> > (*) - also has some differences in the set of class files that are
> included depending on the platform
> >
> > So per-platform versions of the above would be needed to accommodate the
> different native libraries.
> >
> > If it would help, the .class files could be always included for all
> platforms, but there would be some additional effort to make this work.
> Also, it might be just as easy to have the class files and the natives in
> one downloaded jar file per platform, since at least the natives need to be
> platform-independent. I'm far from a maven expert, though, so I don't
> really know for sure which path is easier.
> >
> > -- Kevin
> >
> >
> >> On 4/30/2018 9:44 AM, Paul Ray Russell wrote:
> >>  >I'm not sure I understand the question about platform-specific jar
> files,
> >>
> >> Last time I worked on native specifics (which was to package up RXTX
> dlls
> >> for different OSs / in 64/32 bit) The easiest solution for pure Maven
> >> builds seemed to be, to package DLLs inside a jar. We then used a
> profile
> >> to control which DLL's were grabbed in different cases. And surely for
> this
> >> specific case, it would be better to split the native specifics into
> >> separate jars per OS to avoid a huge cross-OS download?
> >
>
>