Re: Canvas blowing up (was Re: JavaFX Media issues)

[Joseph Andresen]

Canvas as of right now (pretty much any 8.0 build) is actually not 
persistent due to the MT changes that were implemented to make the 
buffer system "work" with multi-threading. I had a patch which 
implements the clearRect (without considering Jim's warning about stale 
state), but it is useless until the buffers are properly handled with 
multiple threads.


Here is the jira for anyone curious:
https://javafx-jira.kenai.com/browse/RT-27558

-Joe


On 8/9/2013 8:23 AM, Richard Bair wrote:

I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to the GC
using a lot more CPU, thus reducing the ability for Canvas to process its
command queue even further, well it just collapses in on itself  and dies.

Forking the thread.

The problem with Canvas is that if you have a canvas and you scribble on it, 
and then scribble on it some more, and then scribble on it some more, then in 
order for us to get the right result in the end, we need to replay all those 
scribbles in order. If pulses are not happening, we still need to remember 
these scribbles so we can draw the right result.

BUT, if you issue a command to the canvas which will cause it to "clear" all 
its contents, then we could throw away any previously buffered data. Right now the only 
way to do that would be a fillRect with a solid fill where the fillRect encompasses the 
entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area.

This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should 
both (under the right conditions) throw away the previously buffered commands. Then all you have to 
do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll 
never buffer more than a single frame's worth of data. We could also add a "clear" method 
which is "clearRect(0, 0, w, h)" to make this more foolproof, and then document it as a 
best practice to clear the canvas before each rendering if you intend to redraw the entire thing on 
each frame.

If you're making use of manually operated "dirty rects" so that you only clear the 
damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till 
kingdom come. So we still need a mechanism exposed in the scene graph of "liveness" and 
associated events so that when the scene is no longer live (for example, when minimized) you could 
stop your animation timer, but for your specific media use case this isn't as important.

Richard

Re: Canvas blowing up (was Re: JavaFX Media issues)

[John Hendrikx]

It was late...I forgot WritableImage is not a Node.

I wrapped it in an ImageView, added some width/height properties that 
react on changes in width/height by replacing the WritableImage, and it 
was almost a drop-in replacement for Canvas for me (the main difference 
being that you have to re-get the PixelWriter after every size change 
since its WritableImage is completely replaced each resize).


The resizing when moving the video from monitor to monitor of different 
sizes worked just as flawless as with Canvas.  I did not encounter any 
memory issues so far (but I only tested for 15 minutes).  I'll keep it 
like this for a while.


--John

Code:

public class ResizableWritableImageView extends ImageView {
  private final DoubleProperty width = new SimpleDoubleProperty();
  public final double getWidth() { return width.get(); }
  public final void setWidth(double width) { this.width.set(width); }
  public final DoubleProperty widthProperty() { return width; }

  private final DoubleProperty height = new SimpleDoubleProperty();
  public final double getHeight() { return height.get(); }
  public final void setHeight(double height) { this.height.set(height); }
  public final DoubleProperty heightProperty() { return height; }

  public ResizableWritableImageView(double w, double h) {
width.addListener(new InvalidationListener() {
  @Override
  public void invalidated(Observable observable) {
updateImageSize();
  }
});

height.addListener(new InvalidationListener() {
  @Override
  public void invalidated(Observable observable) {
updateImageSize();
  }
});

width.set(w);
height.set(h);
  }

  @Override
  public void resize(double w, double h) {
setWidth(w);
setHeight(h);
  }

  public PixelWriter getPixelWriter() {
return ((WritableImage)getImage()).getPixelWriter();
  }

  private void updateImageSize() {
int w = (int)getWidth();
int h = (int)getHeight();

if(w < 1) {
  w = 1;
}
if(h < 1) {
  h = 1;
}

setImage(new WritableImage(w, h));
  }
}

On 10/08/2013 06:07, Richard Bair wrote:

Scale the imageview?

On Aug 9, 2013, at 8:57 PM, John Hendrikx  wrote:


On 10/08/2013 05:46, John Hendrikx wrote:

On 9/08/2013 20:15, Richard Bair wrote:

Also the proposed clear() or empty() option only applies to Canvas correct?
i.e. WritableImages don't suffer from these kind of issues and don't require 
such methods?

That is correct (WritableImage we don't provide a 2D API to use to fill the 
buffer, you just bash the pixels yourself however you like, so we don't have to 
buffer anything up).

Hm, I didn't realize WritableImage had the same kind PixelWriter interface.  
For my usecase, where I just render full frames to a PixelWriter so JavaFX can 
display them in some fashion, why should I not use a WriteableImage instead?

Looking at the docs, I see that one limitation is that the WritableImage cannot 
be resized after construction (something that I do use with Canvas), but I 
think I could work around that by just recreating the Image when its size 
changes... I could just wrap a class around it that does this transparently.

Going to take a look if I can rip out the Canvas code and replace it with a 
resizable WritableImage and see what the results are for full screen video 
playback...

Foiled before I even could get started :/

I need the scaleX and scaleY properties of Canvas as well... basically, my 
input video data is always decoded at native size (say 1920x1080) and then 
scaled down or up to fit the screen.  Since I offer a feature that can switch 
to a different attached monitor at any time, even during playback, I cannot 
tell my video source to scale the video for me (it does not offer a way to 
alter this after playback starts).  I haven't given up entirely on 
WritableImage, but this complicates sufficiently that I cannot just create a 
drop-in replacement for Canvas that suits my usecase.

--John

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Richard Bair]

Scale the imageview?

On Aug 9, 2013, at 8:57 PM, John Hendrikx  wrote:

> On 10/08/2013 05:46, John Hendrikx wrote:
>> On 9/08/2013 20:15, Richard Bair wrote:
 Also the proposed clear() or empty() option only applies to Canvas correct?
 i.e. WritableImages don't suffer from these kind of issues and don't 
 require such methods?
>>> That is correct (WritableImage we don't provide a 2D API to use to fill the 
>>> buffer, you just bash the pixels yourself however you like, so we don't 
>>> have to buffer anything up).
>> Hm, I didn't realize WritableImage had the same kind PixelWriter interface.  
>> For my usecase, where I just render full frames to a PixelWriter so JavaFX 
>> can display them in some fashion, why should I not use a WriteableImage 
>> instead?
>> 
>> Looking at the docs, I see that one limitation is that the WritableImage 
>> cannot be resized after construction (something that I do use with Canvas), 
>> but I think I could work around that by just recreating the Image when its 
>> size changes... I could just wrap a class around it that does this 
>> transparently.
>> 
>> Going to take a look if I can rip out the Canvas code and replace it with a 
>> resizable WritableImage and see what the results are for full screen video 
>> playback...
> Foiled before I even could get started :/
> 
> I need the scaleX and scaleY properties of Canvas as well... basically, my 
> input video data is always decoded at native size (say 1920x1080) and then 
> scaled down or up to fit the screen.  Since I offer a feature that can switch 
> to a different attached monitor at any time, even during playback, I cannot 
> tell my video source to scale the video for me (it does not offer a way to 
> alter this after playback starts).  I haven't given up entirely on 
> WritableImage, but this complicates sufficiently that I cannot just create a 
> drop-in replacement for Canvas that suits my usecase.
> 
> --John
> 
> 

Re: Canvas blowing up (was Re: JavaFX Media issues)

[John Hendrikx]

On 10/08/2013 05:46, John Hendrikx wrote:

On 9/08/2013 20:15, Richard Bair wrote:
Also the proposed clear() or empty() option only applies to Canvas 
correct?
i.e. WritableImages don't suffer from these kind of issues and don't 
require such methods?
That is correct (WritableImage we don't provide a 2D API to use to 
fill the buffer, you just bash the pixels yourself however you like, 
so we don't have to buffer anything up).
Hm, I didn't realize WritableImage had the same kind PixelWriter 
interface.  For my usecase, where I just render full frames to a 
PixelWriter so JavaFX can display them in some fashion, why should I 
not use a WriteableImage instead?


Looking at the docs, I see that one limitation is that the 
WritableImage cannot be resized after construction (something that I 
do use with Canvas), but I think I could work around that by just 
recreating the Image when its size changes... I could just wrap a 
class around it that does this transparently.


Going to take a look if I can rip out the Canvas code and replace it 
with a resizable WritableImage and see what the results are for full 
screen video playback...



Foiled before I even could get started :/

I need the scaleX and scaleY properties of Canvas as well... basically, 
my input video data is always decoded at native size (say 1920x1080) and 
then scaled down or up to fit the screen.  Since I offer a feature that 
can switch to a different attached monitor at any time, even during 
playback, I cannot tell my video source to scale the video for me (it 
does not offer a way to alter this after playback starts).  I haven't 
given up entirely on WritableImage, but this complicates sufficiently 
that I cannot just create a drop-in replacement for Canvas that suits my 
usecase.


--John

Re: Canvas blowing up (was Re: JavaFX Media issues)

[John Hendrikx]

On 9/08/2013 20:15, Richard Bair wrote:

Also the proposed clear() or empty() option only applies to Canvas correct?
i.e. WritableImages don't suffer from these kind of issues and don't require 
such methods?

That is correct (WritableImage we don't provide a 2D API to use to fill the 
buffer, you just bash the pixels yourself however you like, so we don't have to 
buffer anything up).
Hm, I didn't realize WritableImage had the same kind PixelWriter 
interface.  For my usecase, where I just render full frames to a 
PixelWriter so JavaFX can display them in some fashion, why should I not 
use a WriteableImage instead?


Looking at the docs, I see that one limitation is that the WritableImage 
cannot be resized after construction (something that I do use with 
Canvas), but I think I could work around that by just recreating the 
Image when its size changes... I could just wrap a class around it that 
does this transparently.


Going to take a look if I can rip out the Canvas code and replace it 
with a resizable WritableImage and see what the results are for full 
screen video playback...


--John

Re: Canvas blowing up (was Re: JavaFX Media issues)

[John Hendrikx]

On 9/08/2013 17:23, Richard Bair wrote:

I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to the GC
using a lot more CPU, thus reducing the ability for Canvas to process its
command queue even further, well it just collapses in on itself  and dies.

Forking the thread.

The problem with Canvas is that if you have a canvas and you scribble on it, 
and then scribble on it some more, and then scribble on it some more, then in 
order for us to get the right result in the end, we need to replay all those 
scribbles in order. If pulses are not happening, we still need to remember 
these scribbles so we can draw the right result.
That's currently how it works, but I guess for quite a few users it is 
unexpected that a Canvas, which looks like basically a texture where you 
can draw on, has any other (significant) memory requirements apart from 
the initial allocation of the buffer based on its size.  I visualize the 
Canvas as a pixel buffer with low level tools to help you manipulate it, 
and these manipulations occur... well, immediately (like settings bits 
in memory).  Any memory allocated associated with the action you just 
did should disappear as soon as the action is 'applied' to the texture.


Perhaps we'd need more of a flush() style command that forces Canvas to 
apply all outstanding actions now and consolidate them into the texture 
so memory can be freed.

BUT, if you issue a command to the canvas which will cause it to "clear" all 
its contents, then we could throw away any previously buffered data. Right now the only 
way to do that would be a fillRect with a solid fill where the fillRect encompasses the 
entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area.
I think that's definitely a worthwhile solution to investigate.  For my 
use-case it would likely solve any OOM issues, as I can easily "clear" 
the canvas each frame (or better yet, Canvas could notice that the 
PixelWriter I use covers its full area).  Lacking that, I'd prefer to 
have a command that just discards previous actions than requiring an 
unnecessary clear/fill when I know that my next command is going to set 
all pixels anyway.



This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should 
both (under the right conditions) throw away the previously buffered commands. Then all you have to 
do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll 
never buffer more than a single frame's worth of data. We could also add a "clear" method 
which is "clearRect(0, 0, w, h)" to make this more foolproof, and then document it as a 
best practice to clear the canvas before each rendering if you intend to redraw the entire thing on 
each frame.

If you're making use of manually operated "dirty rects" so that you only clear the 
damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till 
kingdom come. So we still need a mechanism exposed in the scene graph of "liveness" and 
associated events so that when the scene is no longer live (for example, when minimized) you could 
stop your animation timer, but for your specific media use case this isn't as important.
For my media use-case (and perhaps Scott's) it would definitely solve 
the issue -- I don't intend to do anything else with Canvas, all other 
things I want to draw will occur in normal JavaFX controls layered 
transparently on top of the Canvas.


I still think though that it is rather unexpected behaviour for Canvas 
that it can use significantly more memory (under certain conditions) 
than just the pre-allocated buffer -- even more so when you use 
PixelWriter -- I really expected that to go straight to the buffer, not 
make yet another copy.


--John

RE: JavaFX Media issues

[John Smith]

> Unfortunately the media team is rather starved for resources these days

Seems to be common to every part of JavaFX development :-(

Thanks for your work on this and for posting the reference to the umbrella 
media input issue.

-Original Message-
From: David DeHaven [mailto:david.deha...@oracle.com] 
Sent: Friday, August 09, 2013 11:38 AM
To: John Smith
Cc: Richard Bair; Felix Bembrick; openjfx-dev@openjdk.java.net List
Subject: Re: JavaFX Media issues

>> So called arbitrary input stream support is among the features for the 
>> future.
> 
> Could somebody post back a reference to the jira, I've seen the feature 
> referred to from other jiras, but I've never been able to find the actual 
> jira reference for this.  Thanks!

Here's an umbrella issue that would cover this:
https://javafx-jira.kenai.com/browse/RT-14938

Unfortunately the media team is rather starved for resources these days (even 
more so than in the past).

-DrD-

Re: JavaFX Media issues

[David DeHaven]

>> So called arbitrary input stream support is among the features for the 
>> future.
> 
> Could somebody post back a reference to the jira, I've seen the feature 
> referred to from other jiras, but I've never been able to find the actual 
> jira reference for this.  Thanks!

Here's an umbrella issue that would cover this:
https://javafx-jira.kenai.com/browse/RT-14938

Unfortunately the media team is rather starved for resources these days (even 
more so than in the past).

-DrD-

RE: JavaFX Media issues

[John Smith]

> So called arbitrary input stream support is among the features for the future.

Could somebody post back a reference to the jira, I've seen the feature 
referred to from other jiras, but I've never been able to find the actual jira 
reference for this.  Thanks!

-Original Message-
From: openjfx-dev-boun...@openjdk.java.net 
[mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Richard Bair
Sent: Friday, August 09, 2013 8:35 AM
To: Felix Bembrick
Cc: openjfx-dev@openjdk.java.net List
Subject: Re: JavaFX Media issues

Forwarded from Kiriil Kirichenko who is getting his mailing list credentials 
sorted out...

>> *From: *Felix Bembrick > <mailto:felix.bembr...@gmail.com>>
>> *Subject: **JavaFX Media issues*
>> *Date: *August 8, 2013 1:31:37 PM PDT
>> *To: *"openjfx-dev@openjdk.java.net
>> <mailto:openjfx-dev@openjdk.java.net> List"
>> mailto:openjfx-dev@openjdk.java.net>>
>> 
>> I am having a look at JavaFX media support and have a couple of questions:
>> 
>> 1. It seems that the only way to load media files is by specifying a 
>> source such as a file path etc.  This is not going to work well for 
>> me as all of my application's content (which includes data, digital 
>> assets, media etc.) is stored in a database on the server and is 
>> loaded through an IO stream.
>> Why doesn't Media support loading of files through a stream?  That 
>> would seem like a common use case to me!

This is correct. So called arbitrary input stream support is among the features 
for the future.

>> 2. I am unable to locate any reference to JavaFX Media in the 
>> JavaDocs for JDK 8 which I found here:
>> 
>> http://download.java.net/jdk8/jfxdocs/index.html
>> 
>> Is this just a glitch?

A glitch related to javadoc execution. FXMedia is still there.

>> 3. Is buffering of media something planned for the future in JavaFX?

What do you mean by buffering ? We have progressive buffer for http links. See 
MediaPlayer.bufferProgressTimeProperty

>> 4. What about live streaming of media content?

We support Http Live Streaming: MpegTS which contains H264 + AAC.

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Richard Bair]

> Also the proposed clear() or empty() option only applies to Canvas correct? 
> i.e. WritableImages don't suffer from these kind of issues and don't require 
> such methods?

That is correct (WritableImage we don't provide a 2D API to use to fill the 
buffer, you just bash the pixels yourself however you like, so we don't have to 
buffer anything up).

Richard

RE: Canvas blowing up (was Re: JavaFX Media issues)

[John Smith]

This question was recently asked on StackOverflow as well:
http://stackoverflow.com/questions/18097404/how-can-i-free-canvas-memory "How 
can I free Canvas memory?"
So others have been running into these kind of issues.

Also the proposed clear() or empty() option only applies to Canvas correct? 
i.e. WritableImages don't suffer from these kind of issues and don't require 
such methods?

Regards,
John

-Original Message-
From: openjfx-dev-boun...@openjdk.java.net 
[mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Richard Bair
Sent: Friday, August 09, 2013 9:43 AM
To: Dr. Michael Paus
Cc: openjfx-dev@openjdk.java.net
Subject: Re: Canvas blowing up (was Re: JavaFX Media issues)

> What would be the performance penalty for using this quick-fix? The 
> clear/fill commands do not just clear the command buffer. They also 
> fill the canvas area with a certain color. So in normal operation the canvas 
> is always filled twice for each frame, isn't it?

That would be correct. Another option is to add, instead of "clear()" an 
explicit "empty()" method or something that would just blow away the buffer.

Richard

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Richard Bair]

> What would be the performance penalty for using this quick-fix? The 
> clear/fill commands do not
> just clear the command buffer. They also fill the canvas area with a certain 
> color. So in normal
> operation the canvas is always filled twice for each frame, isn't it?

That would be correct. Another option is to add, instead of "clear()" an 
explicit "empty()" method or something that would just blow away the buffer.

Richard

Re: Canvas blowing up (was Re: JavaFX Media issues)

[steve . x . northover]

We would still do the fill but we could throw away any buffered commands 
that happened before the fill.


Steve

On 09/08/2013 12:16 PM, Dr. Michael Paus wrote:
What would be the performance penalty for using this quick-fix? The 
clear/fill commands do not
just clear the command buffer. They also fill the canvas area with a 
certain color. So in normal

operation the canvas is always filled twice for each frame, isn't it?


Am 09.08.13 17:23, schrieb Richard Bair:

I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to 
the GC
using a lot more CPU, thus reducing the ability for Canvas to 
process its
command queue even further, well it just collapses in on itself  and 
dies.

Forking the thread.

The problem with Canvas is that if you have a canvas and you scribble 
on it, and then scribble on it some more, and then scribble on it 
some more, then in order for us to get the right result in the end, 
we need to replay all those scribbles in order. If pulses are not 
happening, we still need to remember these scribbles so we can draw 
the right result.


BUT, if you issue a command to the canvas which will cause it to 
"clear" all its contents, then we could throw away any previously 
buffered data. Right now the only way to do that would be a fillRect 
with a solid fill where the fillRect encompasses the entire canvas 
area, or a clearRect where the clearRect encompasses the entire 
canvas area.


This seems like a very simple fix. GraphicsContext.clearRect and 
GraphicsContext.fillRect should both (under the right conditions) 
throw away the previously buffered commands. Then all you have to do 
is be sure to make one of these calls (likely just a clearRect) 
before each frame, and we'll never buffer more than a single frame's 
worth of data. We could also add a "clear" method which is 
"clearRect(0, 0, w, h)" to make this more foolproof, and then 
document it as a best practice to clear the canvas before each 
rendering if you intend to redraw the entire thing on each frame.


If you're making use of manually operated "dirty rects" so that you 
only clear the damaged area to repaint, then we couldn't employ this 
technique and we'd have to buffer 'till kingdom come. So we still 
need a mechanism exposed in the scene graph of "liveness" and 
associated events so that when the scene is no longer live (for 
example, when minimized) you could stop your animation timer, but for 
your specific media use case this isn't as important.


Richard

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Richard Bair]

> That's okay for a quick hack.  In the case of a video preview surface, I will 
> be explicitly setting the value for every pixel from a ByteBuffer.  You could 
> save the extra step of doing a rectFill or clearRect if you knew that every 
> pixel was about to be overwritten.  It's a reasonable optimization.. but as a 
> fix for this issue it's still only a half-fix hack.

I wouldn't call it a hack, just an obvious optimization that would happen to 
help in this case.

> "If pulses are not happening, we still need to remember these scribbles so we 
> can draw the right result."
> 
> No.  If pulses are not happening you need to block or force a pulse somehow. 
> Otherwise I don't see how having the unbounded queue is ever going to be 100% 
> reliable.

I agree it would be better to render the thing. We still need a way to tell 
people that what they're doing isn't visible so they can stop their timer or 
whatnot to avoid wasted CPU cycles / battery drain.

> Since we are talking about painting to the Canvas surface as opposed to 
> directly modifying the scene graph, why does the painting have to happen 
> "later" when a pulse occurs?  It's not like you have any other thread writing 
> to the Canvas.  Why can't the Platform thread actually *do* the scribbles,and 
> the pulse just refreshes the portion of the Canvas that is visible on the 
> screen?  Is it some D3D/OpenGL multi-threading complication?

No doubt drawing to the actual screen in this case won't work (since the 
surface is likely "lost"), but if we're doing double buffering then we should 
still be able to draw to the texture. There might be nasty details to work 
through (windows / graphics guys?). At the high level we short-cut out of doing 
a pulse if the window is not actually visible so we don't waste CPU resources, 
but from a high level it would be easy enough to keep track of whether we have 
dirty canvas' and need to render them anyway. I don't know about the low levels 
whether they would blow up.

Richard

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Scott Palmer]

That's okay for a quick hack.  In the case of a video preview surface, I
will be explicitly setting the value for every pixel from a ByteBuffer.
 You could save the extra step of doing a rectFill or clearRect if you knew
that every pixel was about to be overwritten.  It's a reasonable
optimization.. but as a fix for this issue it's still only a half-fix hack.

"If pulses are not happening, we still need to remember these scribbles so
we can draw the right result."

No.  If pulses are not happening you need to block or force a pulse
somehow. Otherwise I don't see how having the unbounded queue is ever going
to be 100% reliable.

Since we are talking about painting to the Canvas surface as opposed to
directly modifying the scene graph, why does the painting have to happen
"later" when a pulse occurs?  It's not like you have any other thread
writing to the Canvas.  Why can't the Platform thread actually *do* the
scribbles,and the pulse just refreshes the portion of the Canvas that is
visible on the screen?  Is it some D3D/OpenGL multi-threading complication?

Regards,

Scott



On Fri, Aug 9, 2013 at 11:43 AM,  wrote:

> This is a great idea.  We should just enter a tweak and do it.  If the
> area that is being cleared is larger than the current size of the canvas,
> we can throw away all pending draw commands.
>
> Steve
>
>
> On 09/08/2013 11:23 AM, Richard Bair wrote:
>
>> I mean, it looks like it is working for a few seconds,
>>> but then as the memory fills with the Canvas backlog it can lead to the
>>> GC
>>> using a lot more CPU, thus reducing the ability for Canvas to process its
>>> command queue even further, well it just collapses in on itself  and
>>> dies.
>>>
>> Forking the thread.
>>
>> The problem with Canvas is that if you have a canvas and you scribble on
>> it, and then scribble on it some more, and then scribble on it some more,
>> then in order for us to get the right result in the end, we need to replay
>> all those scribbles in order. If pulses are not happening, we still need to
>> remember these scribbles so we can draw the right result.
>>
>> BUT, if you issue a command to the canvas which will cause it to "clear"
>> all its contents, then we could throw away any previously buffered data.
>> Right now the only way to do that would be a fillRect with a solid fill
>> where the fillRect encompasses the entire canvas area, or a clearRect where
>> the clearRect encompasses the entire canvas area.
>>
>> This seems like a very simple fix. GraphicsContext.clearRect and
>> GraphicsContext.fillRect should both (under the right conditions) throw
>> away the previously buffered commands. Then all you have to do is be sure
>> to make one of these calls (likely just a clearRect) before each frame, and
>> we'll never buffer more than a single frame's worth of data. We could also
>> add a "clear" method which is "clearRect(0, 0, w, h)" to make this more
>> foolproof, and then document it as a best practice to clear the canvas
>> before each rendering if you intend to redraw the entire thing on each
>> frame.
>>
>> If you're making use of manually operated "dirty rects" so that you only
>> clear the damaged area to repaint, then we couldn't employ this technique
>> and we'd have to buffer 'till kingdom come. So we still need a mechanism
>> exposed in the scene graph of "liveness" and associated events so that when
>> the scene is no longer live (for example, when minimized) you could stop
>> your animation timer, but for your specific media use case this isn't as
>> important.
>>
>> Richard
>>
>
>

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Dr. Michael Paus]

What would be the performance penalty for using this quick-fix? The 
clear/fill commands do not
just clear the command buffer. They also fill the canvas area with a 
certain color. So in normal

operation the canvas is always filled twice for each frame, isn't it?


Am 09.08.13 17:23, schrieb Richard Bair:

I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to the GC
using a lot more CPU, thus reducing the ability for Canvas to process its
command queue even further, well it just collapses in on itself  and dies.

Forking the thread.

The problem with Canvas is that if you have a canvas and you scribble on it, 
and then scribble on it some more, and then scribble on it some more, then in 
order for us to get the right result in the end, we need to replay all those 
scribbles in order. If pulses are not happening, we still need to remember 
these scribbles so we can draw the right result.

BUT, if you issue a command to the canvas which will cause it to "clear" all 
its contents, then we could throw away any previously buffered data. Right now the only 
way to do that would be a fillRect with a solid fill where the fillRect encompasses the 
entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area.

This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should 
both (under the right conditions) throw away the previously buffered commands. Then all you have to 
do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll 
never buffer more than a single frame's worth of data. We could also add a "clear" method 
which is "clearRect(0, 0, w, h)" to make this more foolproof, and then document it as a 
best practice to clear the canvas before each rendering if you intend to redraw the entire thing on 
each frame.

If you're making use of manually operated "dirty rects" so that you only clear the 
damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till 
kingdom come. So we still need a mechanism exposed in the scene graph of "liveness" and 
associated events so that when the scene is no longer live (for example, when minimized) you could 
stop your animation timer, but for your specific media use case this isn't as important.

Richard

Re: Canvas blowing up (was Re: JavaFX Media issues)

[steve . x . northover]

This is a great idea.  We should just enter a tweak and do it.  If the 
area that is being cleared is larger than the current size of the 
canvas, we can throw away all pending draw commands.


Steve

On 09/08/2013 11:23 AM, Richard Bair wrote:

I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to the GC
using a lot more CPU, thus reducing the ability for Canvas to process its
command queue even further, well it just collapses in on itself  and dies.

Forking the thread.

The problem with Canvas is that if you have a canvas and you scribble on it, 
and then scribble on it some more, and then scribble on it some more, then in 
order for us to get the right result in the end, we need to replay all those 
scribbles in order. If pulses are not happening, we still need to remember 
these scribbles so we can draw the right result.

BUT, if you issue a command to the canvas which will cause it to "clear" all 
its contents, then we could throw away any previously buffered data. Right now the only 
way to do that would be a fillRect with a solid fill where the fillRect encompasses the 
entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area.

This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should 
both (under the right conditions) throw away the previously buffered commands. Then all you have to 
do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll 
never buffer more than a single frame's worth of data. We could also add a "clear" method 
which is "clearRect(0, 0, w, h)" to make this more foolproof, and then document it as a 
best practice to clear the canvas before each rendering if you intend to redraw the entire thing on 
each frame.

If you're making use of manually operated "dirty rects" so that you only clear the 
damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till 
kingdom come. So we still need a mechanism exposed in the scene graph of "liveness" and 
associated events so that when the scene is no longer live (for example, when minimized) you could 
stop your animation timer, but for your specific media use case this isn't as important.

Richard

Re: JavaFX Media issues

[Richard Bair]

Forwarded from Kiriil Kirichenko who is getting his mailing list credentials 
sorted out...

>> *From: *Felix Bembrick > >
>> *Subject: **JavaFX Media issues*
>> *Date: *August 8, 2013 1:31:37 PM PDT
>> *To: *"openjfx-dev@openjdk.java.net
>>  List"
>> mailto:openjfx-dev@openjdk.java.net>>
>> 
>> I am having a look at JavaFX media support and have a couple of questions:
>> 
>> 1. It seems that the only way to load media files is by specifying a
>> source
>> such as a file path etc.  This is not going to work well for me as all of
>> my application's content (which includes data, digital assets, media etc.)
>> is stored in a database on the server and is loaded through an IO stream.
>> Why doesn't Media support loading of files through a stream?  That would
>> seem like a common use case to me!

This is correct. So called arbitrary input stream support is among the features 
for the future.

>> 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for
>> JDK 8 which I found here:
>> 
>> http://download.java.net/jdk8/jfxdocs/index.html
>> 
>> Is this just a glitch?

A glitch related to javadoc execution. FXMedia is still there.

>> 3. Is buffering of media something planned for the future in JavaFX?

What do you mean by buffering ? We have progressive buffer for http links. See 
MediaPlayer.bufferProgressTimeProperty

>> 4. What about live streaming of media content?

We support Http Live Streaming: MpegTS which contains H264 + AAC.

Re: Canvas blowing up (was Re: JavaFX Media issues)

[Richard Bair]

https://javafx-jira.kenai.com/browse/RT-32242

On Aug 9, 2013, at 8:23 AM, Richard Bair  wrote:

>> I mean, it looks like it is working for a few seconds,
>> but then as the memory fills with the Canvas backlog it can lead to the GC
>> using a lot more CPU, thus reducing the ability for Canvas to process its
>> command queue even further, well it just collapses in on itself  and dies.
> 
> Forking the thread.
> 
> The problem with Canvas is that if you have a canvas and you scribble on it, 
> and then scribble on it some more, and then scribble on it some more, then in 
> order for us to get the right result in the end, we need to replay all those 
> scribbles in order. If pulses are not happening, we still need to remember 
> these scribbles so we can draw the right result.
> 
> BUT, if you issue a command to the canvas which will cause it to "clear" all 
> its contents, then we could throw away any previously buffered data. Right 
> now the only way to do that would be a fillRect with a solid fill where the 
> fillRect encompasses the entire canvas area, or a clearRect where the 
> clearRect encompasses the entire canvas area.
> 
> This seems like a very simple fix. GraphicsContext.clearRect and 
> GraphicsContext.fillRect should both (under the right conditions) throw away 
> the previously buffered commands. Then all you have to do is be sure to make 
> one of these calls (likely just a clearRect) before each frame, and we'll 
> never buffer more than a single frame's worth of data. We could also add a 
> "clear" method which is "clearRect(0, 0, w, h)" to make this more foolproof, 
> and then document it as a best practice to clear the canvas before each 
> rendering if you intend to redraw the entire thing on each frame.
> 
> If you're making use of manually operated "dirty rects" so that you only 
> clear the damaged area to repaint, then we couldn't employ this technique and 
> we'd have to buffer 'till kingdom come. So we still need a mechanism exposed 
> in the scene graph of "liveness" and associated events so that when the scene 
> is no longer live (for example, when minimized) you could stop your animation 
> timer, but for your specific media use case this isn't as important.
> 
> Richard

Canvas blowing up (was Re: JavaFX Media issues)

[Richard Bair]

> I mean, it looks like it is working for a few seconds,
> but then as the memory fills with the Canvas backlog it can lead to the GC
> using a lot more CPU, thus reducing the ability for Canvas to process its
> command queue even further, well it just collapses in on itself  and dies.

Forking the thread.

The problem with Canvas is that if you have a canvas and you scribble on it, 
and then scribble on it some more, and then scribble on it some more, then in 
order for us to get the right result in the end, we need to replay all those 
scribbles in order. If pulses are not happening, we still need to remember 
these scribbles so we can draw the right result.

BUT, if you issue a command to the canvas which will cause it to "clear" all 
its contents, then we could throw away any previously buffered data. Right now 
the only way to do that would be a fillRect with a solid fill where the 
fillRect encompasses the entire canvas area, or a clearRect where the clearRect 
encompasses the entire canvas area.

This seems like a very simple fix. GraphicsContext.clearRect and 
GraphicsContext.fillRect should both (under the right conditions) throw away 
the previously buffered commands. Then all you have to do is be sure to make 
one of these calls (likely just a clearRect) before each frame, and we'll never 
buffer more than a single frame's worth of data. We could also add a "clear" 
method which is "clearRect(0, 0, w, h)" to make this more foolproof, and then 
document it as a best practice to clear the canvas before each rendering if you 
intend to redraw the entire thing on each frame.

If you're making use of manually operated "dirty rects" so that you only clear 
the damaged area to repaint, then we couldn't employ this technique and we'd 
have to buffer 'till kingdom come. So we still need a mechanism exposed in the 
scene graph of "liveness" and associated events so that when the scene is no 
longer live (for example, when minimized) you could stop your animation timer, 
but for your specific media use case this isn't as important.

Richard

Re: JavaFX Media issues

[John Hendrikx]

On 9/08/2013 03:48, Scott Palmer wrote:
I have heard rumors of people being able to play HD video via Canvas. 
 I have tried everything and can't come close. (Yes, I have been 
careful about the pixel format.)  I mean, it looks like it is working 
for a few seconds, but then as the memory fills with the Canvas 
backlog it can lead to the GC using a lot more CPU, thus reducing the 
ability for Canvas to process its command queue even further, well it 
just collapses in on itself  and dies.
Is your app able to do *anything* else while the video is playing? 
 The slightest delay to the rendering and that Canvas buffering bug 
kills the app.  Not that it would matter if it could keep up, because 
the off-screen thing is also a deal breaker.
My app is basically just a video watching system -- while video is 
playing (in the bottom most layer of a StackPane), it shows overlays 
with time / position, possibly a menu (where you can download a matching 
subtitle for the video and adjust settings like playback speed and 
brightness).  These overlays are smoothly faded in / faded out just with 
the opacity property of another Pane that is at a higher level in the 
main StackPane.


None of this is really CPU intensive, nor is there a huge SceneGraph to 
deal with (I'm guessing its smaller than 100 nodes while video playback 
is running).  See picture here for an idea of what's going on while 
playback is occuring: 
https://github.com/hjohn/MediaSystem/blob/master/screenshot-1.jpg


The Stage used is a non-transparent one -- I mention this because a 
transparent change performs a lot worse than a non-transparent one.  
Before I used the Canvas solution, I'd actually just playback video in a 
java.awt.Frame with a transparent JavaFX Stage on top.   I'd have 2 
windows, with the transparent JavaFX Stage on top that would show 
overlays for the video, while the other window would show the video 
using java.awt.Canvas, accessed directly by VLC.


I do run into this issue every now and then though: 
https://javafx-jira.kenai.com/browse/RT-23312


It really needs to be fixed or Canvas is simply not safely usable in its 
current form (and IMHO never has been since its release then).


Of course 25fps is well below the 60fps required for full-speed video. 
 I suspect it is the frame rate more than the frame size that matters 
here.  Plain old, standard definition, interlaced, 60 fields per 
second will probably kill most apps with the current Canvas 
implementation.
I don't think I have aany 60fps video, if you can point me to something 
I can download that VLC can handle that is of the size you're using I 
could test with the setup I have here.  With 1920x1080 HD Video, the CPU 
uses 8% according to Task Manager (low-end quad core xeon, about 1 year 
old).  I'm running a standard Java VM (b99), no other tweaking, with 
default memory settings, 256 MB heap, Task Manager claims a working set 
of  +/- 600 MB, but some native memory might be involved -- when 
playback stops, working set drops to 340 MB, so there's definitely a lot 
going on.


It's solid though once playback starts (usually it only locks up at the 
start of playback, if it locks up) and can run for hours.  No frame 
stuttering (I'd notice this immediately on a horizontal pan of one of 
the test videos I use).  Even with a lot of other things going on on the 
same machine (although not by my Java process) playback stays solid -- I 
often have this machine running at 50-75% CPU for extended periods while 
enjoying a Video on the 3rd screen.


I'd agree though that more than doubling the frame rate is gonna be a 
huge impact.  I'm not certain if interlaced is going to cause any 
additional problems, I'm assuming that's handled by VLC and that I'd 
still be copying a full buffer for each frame (or do you mean 60 FPS 
interlaced = 120 FPS?).


We need something better.  I proposed having at least a JNI method so 
we could get native window handles from Stages but didn't get any 
traction either.  Security was brought up as a concern, which I 
totally do NOT understand as the use of JNI means you are out of the 
Java sandbox already.  If we had native window handles we might be 
able get our own window for rendering to at least interact nicely with 
the Stage.  (We already did this successfully in Swing via JAWT and a 
heavyweight component that we paint to from native code)
Getting a WindowHandle for a Stage would be great -- however, I think I 
actually hacked this in at one point, and then tried playing back video 
on a JavaFX Stage -- the video however always ended up in the 
foreground, obliterating any JavaFX rendering.  That would need to be 
solved as well if it is still an issue.


I've actually tried almost every video player solution that I could find 
(all on Windows), including DirectShow (using DSJ), GStreamer-java, 
MPlayer (in seperate window), VLCJ, MediaPlayer Classic Home Cinema and 
Xuggler.  All of them except VLCJ had severe issues, ranging from

Re: JavaFX Media issues

[Scott Palmer]

At minimum the Media Player must support H.264, MP3, AAC (so there is a
guarantee of a cross-platform format) and then defer to the default
platform media decoder for everything else.   I.e. *anything* that the
standard OS provided media system can decode - I.e. QuickTime on Mac,
GStreamer on Linux, DirectShow on Windows.   But all that is needed is the
hook and the community can make the bindings to QuickTime, VLC, etc.

The current implementation appears to be too unstable to rely on at the
moment anyway though: https://javafx-jira.kenai.com/browse/RT-31412

I have heard rumors of people being able to play HD video via Canvas.  I
have tried everything and can't come close. (Yes, I have been careful about
the pixel format.)  I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to the GC
using a lot more CPU, thus reducing the ability for Canvas to process its
command queue even further, well it just collapses in on itself  and dies.
Is your app able to do *anything* else while the video is playing?  The
slightest delay to the rendering and that Canvas buffering bug kills the
app.  Not that it would matter if it could keep up, because the off-screen
thing is also a deal breaker.

Of course 25fps is well below the 60fps required for full-speed video.  I
suspect it is the frame rate more than the frame size that matters here.
 Plain old, standard definition, interlaced, 60 fields per second will
probably kill most apps with the current Canvas implementation.

We need something better.  I proposed having at least a JNI method so we
could get native window handles from Stages but didn't get any traction
either.  Security was brought up as a concern, which I totally do NOT
understand as the use of JNI means you are out of the Java sandbox already.
 If we had native window handles we might be able get our own window for
rendering to at least interact nicely with the Stage.  (We already did this
successfully in Swing via JAWT and a heavyweight component that we paint to
from native code)

Regards,

Scott



On Thu, Aug 8, 2013 at 9:06 PM, John Hendrikx  wrote:

> On 9/08/2013 02:10, Scott Palmer wrote:
>
>> The Media APIs are mostly useless in their current state.  Other than
>> demoing that you can play a video, they don't go far enough to be of
>> practical value.  I tried to get someone to pay attention to them back in
>> the JavaFX 1.0 days
>> https://javafx-jira.kenai.com/**browse/RT-2684
>>
>
> Unfortunately, I have to agree here.  I'd love to use the Media APIs of
> JavaFX, but they are way to limited.  At a minimum I'd need support for MKV
> containers and some common audio formats used with these.  Decoding AVI
> containers would be rather important as well.  Without those, JavaFX video
> is basically limited to "bring your own" videos only and completely
> unsuitable for playing back videos that end users might have...
>
> Instead, I've been using VLC + VLCJ from day one while working with JavaFX
> as a work-around.
>
>
>  at least someone listened to the request to get H.264 support in there,
>> but
>> that is just a workaround.  We need to be able to get our data into the
>> media pipeline.  This would allow those of us that have attempted to do a
>> video window to have a fighting chance.  Canvas can't keep up and will
>> likely crash the app with out of memory errors.  Support for drawing into
>> a
>> native surface (OpenGL or D3D context) has been talked about, but doesn't
>> appear to be on the horizon yet.  If we just had a hook to get the dang
>> pixel data into the media pipeline so we could supply the "next frame"
>> with
>> whatever we want - either from any native codec via JNI, or dynamically
>> generated from Java code, whatever... that would be just so dang useful...
>> (to me at least)
>>
> This would atleast allow me to decode the MKV container myself and supply
> data... not looking forward to having to write all that code, but I would
> if it meant I'd be able to go "native" JavaFX with video.
>
> Canvas however is working for me even with HD video (copying 25 frames/sec
> of HD video from VLCJ to Canvas).  Playback is smooth even with 1920x1080
> video for hours long video, and I can't tell the difference with a native
> player or copying frame by frame using pixelWriter anymore.  There is some
> CPU penalty but on a my system it is well below 5%.  This is basically how
> that looks with VLCJ:
>
> new RenderCallback() {
>   @Override
>   public void display(DirectMediaPlayer mp, Memory[] memory, final
> BufferFormat bufferFormat) {
> final ByteBuffer byteBuffer = memory[0].getByteBuffer(0,
> memory[0].size());
>
> Platform.runLater(new Runnable() {
>   @Override
>   public void run() {
> pixelWriter.setPixels(0, 0, bufferFormat.getWidth(),
> bufferFormat.getHeight(), byteBgraInstance, b

Re: JavaFX Media issues

[John Hendrikx]

On 9/08/2013 02:10, Scott Palmer wrote:

The Media APIs are mostly useless in their current state.  Other than
demoing that you can play a video, they don't go far enough to be of
practical value.  I tried to get someone to pay attention to them back in
the JavaFX 1.0 days
https://javafx-jira.kenai.com/browse/RT-2684


Unfortunately, I have to agree here.  I'd love to use the Media APIs of 
JavaFX, but they are way to limited.  At a minimum I'd need support for 
MKV containers and some common audio formats used with these.  Decoding 
AVI containers would be rather important as well.  Without those, JavaFX 
video is basically limited to "bring your own" videos only and 
completely unsuitable for playing back videos that end users might have...


Instead, I've been using VLC + VLCJ from day one while working with 
JavaFX as a work-around.



at least someone listened to the request to get H.264 support in there, but
that is just a workaround.  We need to be able to get our data into the
media pipeline.  This would allow those of us that have attempted to do a
video window to have a fighting chance.  Canvas can't keep up and will
likely crash the app with out of memory errors.  Support for drawing into a
native surface (OpenGL or D3D context) has been talked about, but doesn't
appear to be on the horizon yet.  If we just had a hook to get the dang
pixel data into the media pipeline so we could supply the "next frame" with
whatever we want - either from any native codec via JNI, or dynamically
generated from Java code, whatever... that would be just so dang useful...
(to me at least)
This would atleast allow me to decode the MKV container myself and 
supply data... not looking forward to having to write all that code, but 
I would if it meant I'd be able to go "native" JavaFX with video.


Canvas however is working for me even with HD video (copying 25 
frames/sec of HD video from VLCJ to Canvas).  Playback is smooth even 
with 1920x1080 video for hours long video, and I can't tell the 
difference with a native player or copying frame by frame using 
pixelWriter anymore.  There is some CPU penalty but on a my system it is 
well below 5%.  This is basically how that looks with VLCJ:


new RenderCallback() {
  @Override
  public void display(DirectMediaPlayer mp, Memory[] memory, 
final BufferFormat bufferFormat) {
final ByteBuffer byteBuffer = memory[0].getByteBuffer(0, 
memory[0].size());


Platform.runLater(new Runnable() {
  @Override
  public void run() {
pixelWriter.setPixels(0, 0, bufferFormat.getWidth(), 
bufferFormat.getHeight(), byteBgraInstance, byteBuffer, 
bufferFormat.getPitches()[0]);

  }
});
  }
}

However, the bug where Canvas keeps buffering data when off-screen 
sometimes bites me, resulting in out of memory errors -- even slight 
display delays can fairly easily cause this when video is running at 25 
fps (takes about a second to go through 256 MB of memory).


--John



Regards,

Scott



On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici<
fabrizio.giud...@tidalwave.it>  wrote:


On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn
wrote:

  I don't know why FX Media isn't in the FX 8 API docs, but that's clearly

an error.  Please file a bug on that.

In the meantime, you should look at the FX 2 media docs, there isn't a
lot of change from FX2 media in FX8.  Buffering and streaming (HTTP Live
Streaming) are both supported, as is playback from a URL.


What is the strategy for codecs? I mean, now we have ImageIO (there is
also JAI but it seems basically dead). ImageIO provides many image codecs
and there's a SPI that can be used to support more formats. Will it be
replaced by FX2 media or co-exist with it?


--
Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
"We make Java work. Everywhere."
http://tidalwave.it/fabrizio/**blog  -
fabrizio.giud...@tidalwave.it

Re: JavaFX Media issues

[Scott Palmer]

I was referring to supplying the pixel data to the media nodes.  Getting
the pixel data of the media from the media nodes should also be possible.
 It actually is possible now to a degree via the "snapshot" method on Node.
 Though I encountered a lot of lock-up problems when I tried it.

Scott


On Thu, Aug 8, 2013 at 8:15 PM, Felix Bembrick wrote:

> Scott,
>
> That JIRA issue doesn't make any mention of getting hooks into pixel data
> etc., it only refers to user defined codecs.
>
> Maybe you could open another issue to manage the wishlist of missing
> features such as the ones you referred to?  I too would very much like to
> see greatly improved media support in JavaFX.
>
> Felix
>
>
> On 9 August 2013 10:10, Scott Palmer  wrote:
>
>> The Media APIs are mostly useless in their current state.  Other than
>> demoing that you can play a video, they don't go far enough to be of
>> practical value.  I tried to get someone to pay attention to them back in
>> the JavaFX 1.0 days
>> https://javafx-jira.kenai.com/browse/RT-2684
>>
>> at least someone listened to the request to get H.264 support in there,
>> but that is just a workaround.  We need to be able to get our data into the
>> media pipeline.  This would allow those of us that have attempted to do a
>> video window to have a fighting chance.  Canvas can't keep up and will
>> likely crash the app with out of memory errors.  Support for drawing into a
>> native surface (OpenGL or D3D context) has been talked about, but doesn't
>> appear to be on the horizon yet.  If we just had a hook to get the dang
>> pixel data into the media pipeline so we could supply the "next frame" with
>> whatever we want - either from any native codec via JNI, or dynamically
>> generated from Java code, whatever... that would be just so dang useful...
>> (to me at least)
>>
>> Regards,
>>
>> Scott
>>
>>
>>
>> On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici <
>> fabrizio.giud...@tidalwave.it> wrote:
>>
>>> On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn 
>>> wrote:
>>>
>>>  I don't know why FX Media isn't in the FX 8 API docs, but that's
 clearly an error.  Please file a bug on that.

 In the meantime, you should look at the FX 2 media docs, there isn't a
 lot of change from FX2 media in FX8.  Buffering and streaming (HTTP Live
 Streaming) are both supported, as is playback from a URL.

>>>
>>> What is the strategy for codecs? I mean, now we have ImageIO (there is
>>> also JAI but it seems basically dead). ImageIO provides many image codecs
>>> and there's a SPI that can be used to support more formats. Will it be
>>> replaced by FX2 media or co-exist with it?
>>>
>>>
>>> --
>>> Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
>>> "We make Java work. Everywhere."
>>> http://tidalwave.it/fabrizio/**blog -
>>> fabrizio.giud...@tidalwave.it
>>>
>>
>>
>

Re: JavaFX Media issues

[Daniel Zwolenski]

I don't want to detract from the issues around media stuff (they are
significant), but if you are desperate (as I was), here's some code for
doing video capture and streaming based on LTI-CIVIL and writing to JFX
image to render the video:

   - https://code.google.com/p/jfxcamera/

It's a work around and not elegant at all, and media needs to have a lot
done to be useful. I only include it here in case there is something useful
you can take from it to hobble through.



On Fri, Aug 9, 2013 at 10:10 AM, Scott Palmer  wrote:

> The Media APIs are mostly useless in their current state.  Other than
> demoing that you can play a video, they don't go far enough to be of
> practical value.  I tried to get someone to pay attention to them back in
> the JavaFX 1.0 days
> https://javafx-jira.kenai.com/browse/RT-2684
>
> at least someone listened to the request to get H.264 support in there, but
> that is just a workaround.  We need to be able to get our data into the
> media pipeline.  This would allow those of us that have attempted to do a
> video window to have a fighting chance.  Canvas can't keep up and will
> likely crash the app with out of memory errors.  Support for drawing into a
> native surface (OpenGL or D3D context) has been talked about, but doesn't
> appear to be on the horizon yet.  If we just had a hook to get the dang
> pixel data into the media pipeline so we could supply the "next frame" with
> whatever we want - either from any native codec via JNI, or dynamically
> generated from Java code, whatever... that would be just so dang useful...
> (to me at least)
>
> Regards,
>
> Scott
>
>
>
> On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici <
> fabrizio.giud...@tidalwave.it> wrote:
>
> > On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn 
> > wrote:
> >
> >  I don't know why FX Media isn't in the FX 8 API docs, but that's clearly
> >> an error.  Please file a bug on that.
> >>
> >> In the meantime, you should look at the FX 2 media docs, there isn't a
> >> lot of change from FX2 media in FX8.  Buffering and streaming (HTTP Live
> >> Streaming) are both supported, as is playback from a URL.
> >>
> >
> > What is the strategy for codecs? I mean, now we have ImageIO (there is
> > also JAI but it seems basically dead). ImageIO provides many image codecs
> > and there's a SPI that can be used to support more formats. Will it be
> > replaced by FX2 media or co-exist with it?
> >
> >
> > --
> > Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
> > "We make Java work. Everywhere."
> > http://tidalwave.it/fabrizio/**blog 
> -
> > fabrizio.giud...@tidalwave.it
> >
>

Re: JavaFX Media issues

[Scott Palmer]

The Media APIs are mostly useless in their current state.  Other than
demoing that you can play a video, they don't go far enough to be of
practical value.  I tried to get someone to pay attention to them back in
the JavaFX 1.0 days
https://javafx-jira.kenai.com/browse/RT-2684

at least someone listened to the request to get H.264 support in there, but
that is just a workaround.  We need to be able to get our data into the
media pipeline.  This would allow those of us that have attempted to do a
video window to have a fighting chance.  Canvas can't keep up and will
likely crash the app with out of memory errors.  Support for drawing into a
native surface (OpenGL or D3D context) has been talked about, but doesn't
appear to be on the horizon yet.  If we just had a hook to get the dang
pixel data into the media pipeline so we could supply the "next frame" with
whatever we want - either from any native codec via JNI, or dynamically
generated from Java code, whatever... that would be just so dang useful...
(to me at least)

Regards,

Scott



On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici <
fabrizio.giud...@tidalwave.it> wrote:

> On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn 
> wrote:
>
>  I don't know why FX Media isn't in the FX 8 API docs, but that's clearly
>> an error.  Please file a bug on that.
>>
>> In the meantime, you should look at the FX 2 media docs, there isn't a
>> lot of change from FX2 media in FX8.  Buffering and streaming (HTTP Live
>> Streaming) are both supported, as is playback from a URL.
>>
>
> What is the strategy for codecs? I mean, now we have ImageIO (there is
> also JAI but it seems basically dead). ImageIO provides many image codecs
> and there's a SPI that can be used to support more formats. Will it be
> replaced by FX2 media or co-exist with it?
>
>
> --
> Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
> "We make Java work. Everywhere."
> http://tidalwave.it/fabrizio/**blog  -
> fabrizio.giud...@tidalwave.it
>

Re: JavaFX Media issues

[Felix Bembrick]

Scott,

That JIRA issue doesn't make any mention of getting hooks into pixel data
etc., it only refers to user defined codecs.

Maybe you could open another issue to manage the wishlist of missing
features such as the ones you referred to?  I too would very much like to
see greatly improved media support in JavaFX.

Felix


On 9 August 2013 10:10, Scott Palmer  wrote:

> The Media APIs are mostly useless in their current state.  Other than
> demoing that you can play a video, they don't go far enough to be of
> practical value.  I tried to get someone to pay attention to them back in
> the JavaFX 1.0 days
> https://javafx-jira.kenai.com/browse/RT-2684
>
> at least someone listened to the request to get H.264 support in there,
> but that is just a workaround.  We need to be able to get our data into the
> media pipeline.  This would allow those of us that have attempted to do a
> video window to have a fighting chance.  Canvas can't keep up and will
> likely crash the app with out of memory errors.  Support for drawing into a
> native surface (OpenGL or D3D context) has been talked about, but doesn't
> appear to be on the horizon yet.  If we just had a hook to get the dang
> pixel data into the media pipeline so we could supply the "next frame" with
> whatever we want - either from any native codec via JNI, or dynamically
> generated from Java code, whatever... that would be just so dang useful...
> (to me at least)
>
> Regards,
>
> Scott
>
>
>
> On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici <
> fabrizio.giud...@tidalwave.it> wrote:
>
>> On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn 
>> wrote:
>>
>>  I don't know why FX Media isn't in the FX 8 API docs, but that's clearly
>>> an error.  Please file a bug on that.
>>>
>>> In the meantime, you should look at the FX 2 media docs, there isn't a
>>> lot of change from FX2 media in FX8.  Buffering and streaming (HTTP Live
>>> Streaming) are both supported, as is playback from a URL.
>>>
>>
>> What is the strategy for codecs? I mean, now we have ImageIO (there is
>> also JAI but it seems basically dead). ImageIO provides many image codecs
>> and there's a SPI that can be used to support more formats. Will it be
>> replaced by FX2 media or co-exist with it?
>>
>>
>> --
>> Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
>> "We make Java work. Everywhere."
>> http://tidalwave.it/fabrizio/**blog -
>> fabrizio.giud...@tidalwave.it
>>
>
>

Re: JavaFX Media issues

[Kevin Rushforth]

There is already bug filed to restore the FX Media APIs to the FX 8 API 
docs:


https://javafx-jira.kenai.com/browse/RT-32004

-- Kevin



Joe McGlynn wrote:

I don't know why FX Media isn't in the FX 8 API docs, but that's clearly an 
error.  Please file a bug on that.

In the meantime, you should look at the FX 2 media docs, there isn't a lot of 
change from FX2 media in FX8.  Buffering and streaming (HTTP Live Streaming) 
are both supported, as is playback from a URL.


On Aug 8, 2013, at 1:31 PM, Felix Bembrick  wrote:

  

I am having a look at JavaFX media support and have a couple of questions:

1. It seems that the only way to load media files is by specifying a source
such as a file path etc.  This is not going to work well for me as all of
my application's content (which includes data, digital assets, media etc.)
is stored in a database on the server and is loaded through an IO stream.
Why doesn't Media support loading of files through a stream?  That would
seem like a common use case to me!

2. I am unable to locate any reference to JavaFX Media in the JavaDocs for
JDK 8 which I found here:

http://download.java.net/jdk8/jfxdocs/index.html

Is this just a glitch?

3. Is buffering of media something planned for the future in JavaFX?

4. What about live streaming of media content?

Thanks,

Felix



  

RE: JavaFX Media issues

[John Smith]

> One thing to suggest is that you can install your own URL content handlers / 
> protocols in Java.

That won't help in this case unless you replaced the http, file or jar protocol 
handlers, which would be weird.
"Only HTTP, FILE, and JAR URLs are supported."
http://docs.oracle.com/javafx/2/api/javafx/scene/media/Media.html#Media%28java.lang.String%29

-Original Message-
From: openjfx-dev-boun...@openjdk.java.net 
[mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Richard Bair
Sent: Thursday, August 08, 2013 1:57 PM
To: Felix Bembrick
Cc: openjfx-dev@openjdk.java.net List
Subject: Re: JavaFX Media issues

> I am having a look at JavaFX media support and have a couple of questions:
> 
> 1. It seems that the only way to load media files is by specifying a 
> source such as a file path etc.  This is not going to work well for me 
> as all of my application's content (which includes data, digital 
> assets, media etc.) is stored in a database on the server and is loaded 
> through an IO stream.
> Why doesn't Media support loading of files through a stream?  That 
> would seem like a common use case to me!

I will have to let the media guys chime in, I don't know why we don't support a 
stream. At one time Image had the same limitation but we added stream support 
for it, so it might just be oversight. One thing to suggest is that you can 
install your own URL content handlers / protocols in Java. So if your URL said 
"myapp://nameofvideo.mov" or whatever, you can install a "myapp" protocol 
handler with Java which is responsible for creating the InputStream. This way 
you can still had us a url and we can get back your input stream, even from a 
custom source such as a Database.

> 2. I am unable to locate any reference to JavaFX Media in the JavaDocs 
> for JDK 8 which I found here:
> 
> http://download.java.net/jdk8/jfxdocs/index.html
> 
> Is this just a glitch?

Probably got blown up during the gradle switch. Can you please file a bug (in 
the "build" component)

> 3. Is buffering of media something planned for the future in JavaFX?
> 
> 4. What about live streaming of media content?

Kirill should have more information here.

Thanks!
Richard

Re: JavaFX Media issues

[Fabrizio Giudici]

On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn   
wrote:


I don't know why FX Media isn't in the FX 8 API docs, but that's clearly  
an error.  Please file a bug on that.


In the meantime, you should look at the FX 2 media docs, there isn't a  
lot of change from FX2 media in FX8.  Buffering and streaming (HTTP Live  
Streaming) are both supported, as is playback from a URL.


What is the strategy for codecs? I mean, now we have ImageIO (there is  
also JAI but it seems basically dead). ImageIO provides many image codecs  
and there's a SPI that can be used to support more formats. Will it be  
replaced by FX2 media or co-exist with it?



--
Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
"We make Java work. Everywhere."
http://tidalwave.it/fabrizio/blog - fabrizio.giud...@tidalwave.it

Re: JavaFX Media issues

[Joe McGlynn]

I don't know why FX Media isn't in the FX 8 API docs, but that's clearly an 
error.  Please file a bug on that.

In the meantime, you should look at the FX 2 media docs, there isn't a lot of 
change from FX2 media in FX8.  Buffering and streaming (HTTP Live Streaming) 
are both supported, as is playback from a URL.


On Aug 8, 2013, at 1:31 PM, Felix Bembrick  wrote:

> I am having a look at JavaFX media support and have a couple of questions:
> 
> 1. It seems that the only way to load media files is by specifying a source
> such as a file path etc.  This is not going to work well for me as all of
> my application's content (which includes data, digital assets, media etc.)
> is stored in a database on the server and is loaded through an IO stream.
> Why doesn't Media support loading of files through a stream?  That would
> seem like a common use case to me!
> 
> 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for
> JDK 8 which I found here:
> 
> http://download.java.net/jdk8/jfxdocs/index.html
> 
> Is this just a glitch?
> 
> 3. Is buffering of media something planned for the future in JavaFX?
> 
> 4. What about live streaming of media content?
> 
> Thanks,
> 
> Felix

Re: JavaFX Media issues

[Richard Bair]

> I am having a look at JavaFX media support and have a couple of questions:
> 
> 1. It seems that the only way to load media files is by specifying a source
> such as a file path etc.  This is not going to work well for me as all of
> my application's content (which includes data, digital assets, media etc.)
> is stored in a database on the server and is loaded through an IO stream.
> Why doesn't Media support loading of files through a stream?  That would
> seem like a common use case to me!

I will have to let the media guys chime in, I don't know why we don't support a 
stream. At one time Image had the same limitation but we added stream support 
for it, so it might just be oversight. One thing to suggest is that you can 
install your own URL content handlers / protocols in Java. So if your URL said 
"myapp://nameofvideo.mov" or whatever, you can install a "myapp" protocol 
handler with Java which is responsible for creating the InputStream. This way 
you can still had us a url and we can get back your input stream, even from a 
custom source such as a Database.

> 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for
> JDK 8 which I found here:
> 
> http://download.java.net/jdk8/jfxdocs/index.html
> 
> Is this just a glitch?

Probably got blown up during the gradle switch. Can you please file a bug (in 
the "build" component)

> 3. Is buffering of media something planned for the future in JavaFX?
> 
> 4. What about live streaming of media content?

Kirill should have more information here.

Thanks!
Richard

RE: JavaFX Media issues

[John Smith]

JavaFX 2.2 does http live streaming: 
http://docs.oracle.com/javafx/2/media/overview.htm

"HTTP Live Streaming Support

With the addition of HTTP live streaming support, you can now download the 
playlist file and playback video or audio segments using JavaFX Media. Media 
players are now able to switch to alternate streams, as specified in the 
playlist file and based on network conditions. For a given stream, there is a 
playlist file and a set of segments into which the stream is broken. The stream 
can be either an MP3 raw stream or an MPEG-TS containing multiplexed AAC audio 
and H.264 video. The stream can be played on demand when the stream is a static 
file or played live when the stream is actually a live feed. In both cases, the 
stream can adjust its bit rate and for video, its resolution can be adjusted."

More info on HTTP Live Streaming:
http://developer.apple.com/library/ios/#documentation/networkinginternet/conceptual/streamingmediaguide/Introduction/Introduction.html

Your other questions, the JavaFX media devs can answer.

-Original Message-
From: openjfx-dev-boun...@openjdk.java.net 
[mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Felix Bembrick
Sent: Thursday, August 08, 2013 1:32 PM
To: openjfx-dev@openjdk.java.net List
Subject: JavaFX Media issues

I am having a look at JavaFX media support and have a couple of questions:

1. It seems that the only way to load media files is by specifying a source 
such as a file path etc.  This is not going to work well for me as all of my 
application's content (which includes data, digital assets, media etc.) is 
stored in a database on the server and is loaded through an IO stream.
 Why doesn't Media support loading of files through a stream?  That would seem 
like a common use case to me!

2. I am unable to locate any reference to JavaFX Media in the JavaDocs for JDK 
8 which I found here:

http://download.java.net/jdk8/jfxdocs/index.html

Is this just a glitch?

3. Is buffering of media something planned for the future in JavaFX?

4. What about live streaming of media content?

Thanks,

Felix