I don't want to hijack the WebGL discussion but since it rolled into the 3D library territory anyway I'll give my 2 cents.
3D enhancement are indeed not planned for Java10 (at the minimum) and indeed you can't bring your own shader (asked already at https://stackoverflow.com/questions/43622856/can-we-implement-our-own-materials-in-javafx), but I agree with Mike - you can, maybe somewhat surprisingly, do quite a lot with what there is. Perhaps the most limiting feature is not supporting industry standards of 3D modeling via converters (import/export). It has been suggested ( https://bugs.openjdk.java.net/browse/JDK-8091851) but last activity was 5 years ago. As for shaders (materials), lightings etc., from what I remember by looking around in the source, it will take some effort to rewrite the API to be able to accept custom ones but it's far from impossible. If Phong is implemented there's little reason reason others won't fit (maybe reflective surfaces don't work). Similarly a directional light can be based on the implemented point light be using a cone instead of a sphere. We've employed some clever tricks to get adequate "advanced features" results and considering that all of it can be single-handedly run on iOS and Android with Gluon Mobile (specifically JavaFXPorts) I think there *is* a future in this direction and I'm willing to team up with whomever is interested provided we can get minimal support from the Oracle team.
