Re: [Red5] About the Fix of FLV and FLVReader in regards with OutOfMemoryError when playing large FLVs

[Daniel Daley]

I hacked together a quick version of FLVReader that eliminates the  
memory mapping and pulls straight from the FileInputStream channel  
and it so far has been able to handle much much more. The load on the  
server is terribly high of course. Is this something that could be  
implemented in the server as an option? It would probably be better  
load wise if I created a buffer of a fixed size (couple hundred K or  
something) and it would just keep that buffer filled with data from  
the file as it needed it. I'll give that a shot if I get a chance but  
having something like this in the actual Red5 distribution would be  
wonderful, as loading the whole file to memory puts way too small of  
a cap on how much we can serve.

Thanks,
--Dan--


On Nov 21, 2006, at 1:19 PM, Daniel Daley wrote:

Looking at the code I guess it does load the file into memory  
completely.

channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());

Wouldn't it be more efficient/scalable to have it seek around the  
FileInputStream and read the appropriate bytes for header data etc,  
and then have a configurable sized ByteBuffer for the reading of  
the main file data?

Thanks,

--Dan--


On Nov 21, 2006, at 12:09 PM, Daniel Daley wrote:

What I think I'm running into here is the 2 gig per process memory  
limit of 32 bit processors. Is there any way to reduce the amount  
of memory required per connection? Right now it is still acting as  
if it loads the entire file into memory as our files are on  
average 20 megs and we can get up to around 100 connections before  
the crash. Our server has 4 gigs of ram in it so we have plenty  
left to use. I don't understand the java threading system very  
well, but shouldn't the threading that red5 does create a separate  
process with it's own limits per connection? Or do all connections  
share the single master red5 process?

Thanks,

--Dan--

On Nov 21, 2006, at 11:23 AM, Daniel Daley wrote:

I spoke too soon. Once the connections starting picking up today  
we went back to the previous errors. This one was at 100  
connected viewers:

2006-11-21 13:17:59.572534500 [ERROR] 194490 pool-1-thread-4: 
( org.red5.io.flv.impl.FLVReader.error ) FLVReader :: FLVReader ::
>
2006-11-21 13:17:59.572539500
2006-11-21 13:17:59.572541500 java.io.IOException: Cannot  
allocate memory
2006-11-21 13:17:59.572543500   at sun.nio.ch.FileChannelImpl.map0 
(Native Method)
2006-11-21 13:17:59.572545500   at sun.nio.ch.FileChannelImpl.map 
(FileChannelImpl.java:742)
2006-11-21 13:17:59.572548500   at  
org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
2006-11-21 13:17:59.572550500   at  
org.red5.io.flv.impl.FLV.getReader(FLV.java:172)
2006-11-21 13:17:59.572733500   at  
org.red5.server.stream.provider.FileProvider.init 
(FileProvider.java:179)
2006-11-21 13:17:59.572737500   at  
org.red5.server.stream.provider.FileProvider.pullMessage 
(FileProvider.java:87)
2006-11-21 13:17:59.572739500   at  
org.red5.server.messaging.InMemoryPullPullPipe.pullMessage 
(InMemoryPullPullPipe.java:72)
2006-11-21 13:17:59.572742500   at  
org.red5.server.stream.PlaylistSubscriberStream 
$PlayEngine.pullAndPush(PlaylistSubscriberStr
eam.java:869)
2006-11-21 13:17:59.572789500   at  
org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play 
(PlaylistSubscriberStream.jav
a:703)
2006-11-21 13:17:59.572792500   at  
org.red5.server.stream.PlaylistSubscriberStream.play 
(PlaylistSubscriberStream.java:127)
2006-11-21 13:17:59.572795500   at  
org.red5.server.stream.StreamService.play(StreamService.java:175)
2006-11-21 13:17:59.572797500   at  
org.red5.server.stream.StreamService.play(StreamService.java:183)
2006-11-21 13:17:59.572821500   at  
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
2006-11-21 13:17:59.572823500   at  
sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)
2006-11-21 13:17:59.572826500   at java.lang.reflect.Method.invoke 
(Method.java:585)
2006-11-21 13:17:59.572828500   at  
org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java: 
161)
2006-11-21 13:17:59.572830500   at  
org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java: 
287)
2006-11-21 13:17:59.572840500   at  
org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:469)
2006-11-21 13:17:59.572842500   at  
org.red5.server.net.rtmp.RTMPHandler.messageReceived 
(RTMPHandler.java:147)
2006-11-21 13:17:59.572844500   at  
org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived 
(RTMPMinaIoHandler.java:78)
2006-11-21 13:17:59.572847500   at  
org.apache.mina.common.support.AbstractIoFilterChain 
$2.messageReceived(AbstractIoFilterChain
.java:189)
2006-11-21 13:17:59.572856500   at  
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessage 
Received(AbstractIoFilte
rChain.java:502)
2006-11-21 13:17:59.572858500   at  
org.apache.mina.common.support.AbstractIoFilterChain.access$1000 
(AbstractIoFilterChain.java:
52)
2006-11-21 13:17:59.572861500   at  
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl 
$1.messageReceived(AbstractIoF
ilterChain.java:777)
2006-11-21 13:17:59.572876500   at  
org.red5.io.filter.ExecutorFilter.processEvent 
(ExecutorFilter.java:231)
2006-11-21 13:17:59.572879500   at  
org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run 
(ExecutorFilter.java:279)
2006-11-21 13:17:59.572882500   at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask 
(ThreadPoolExecutor.java:650)
2006-11-21 13:17:59.572885500   at  
java.util.concurrent.ThreadPoolExecutor$Worker.run 
(ThreadPoolExecutor.java:675)
2006-11-21 13:17:59.572952500   at java.lang.Thread.run 
(Thread.java:595)


On Nov 20, 2006, at 5:58 PM, Steven Gong wrote:



On 11/21/06, Daniel Daley <[EMAIL PROTECTED]> wrote:
Looks like it actually just crashes when it reaches whatever amount
of ram the process is limited to. Being 32 bit if I leave all  
limits
off once red5 allocates 2 gigs it crashes. Oddly though since this
patch the ram just seems to grow endlessly until it dies rather  
than
fluctuating like it did before. Could there be something that is  
not
releasing memory after each connection?

It's quite strange that your result is different from that of  
Bill's.

Could you please try to disable the buffer by setting the type  
as "none"? And use NoCacheImpl please.

Thanks,

--Dan--


On Nov 20, 2006, at 3:01 PM, Daniel Daley wrote:

> Looks like mine is crashing much more frequently now with this  
patch
> but with the same errors each time. What's odd is the server  
does not
> in the least seem to be out of memory when it happens. Here's the
> output in my log:
>
> 2006-11-20 16:58:39.560730500 [ERROR] 437909 pool-1-thread-4:
> ( org.red5.io.flv.impl.FLVReader.error ) FLVReader ::  
FLVReader ::>
> 2006-11-20 16:58:39.560735500
> 2006-11-20 16:58:39.560737500 java.io.IOException: Cannot  
allocate
> memory
> 2006-11-20 16:58:39.560739500   at  
sun.nio.ch.FileChannelImpl.map0
> (Native Method)
> 2006-11-20 16:58:39.560741500    at  
sun.nio.ch.FileChannelImpl.map
> (FileChannelImpl.java:742)
> 2006-11-20 16:58:39.560744500   at
> org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
> 2006-11-20 16:58:39.560746500    at  
org.red5.io.flv.impl.FLV.getReader
> (FLV.java:172)
> 2006-11-20 16:58:39.560917500   at
> org.red5.server.stream.provider.FileProvider.init 
(FileProvider.java:
> 179)
> 2006-11-20 16:58: 39.560920500   at
> org.red5.server.stream.provider.FileProvider.pullMessage
> (FileProvider.java:87)
> 2006-11-20 16:58:39.560922500   at
> org.red5.server.messaging.InMemoryPullPullPipe.pullMessage
> (InMemoryPullPullPipe.java:72)
> 2006-11-20 16:58:39.560925500   at
> org.red5.server.stream.PlaylistSubscriberStream 
$PlayEngine.pullAndPush
> (PlaylistSubscriberStream.java:869)
> 2006-11-20 16:58: 39.560937500   at
> org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play
> (PlaylistSubscriberStream.java:703)
> 2006-11-20 16:58:39.560940500   at
> org.red5.server.stream.PlaylistSubscriberStream.play
> (PlaylistSubscriberStream.java:127)
> 2006-11-20 16:58:39.560943500   at
> org.red5.server.stream.StreamService.play(StreamService.java:175)
> 2006-11-20 16:58:39.560945500   at
> org.red5.server.stream.StreamService.play (StreamService.java: 
183)
> 2006-11-20 16:58:39.560960500   at
> sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> 2006-11-20 16:58:39.560962500   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> 2006-11-20 16:58:39.560965500   at  
java.lang.reflect.Method.invoke
> (Method.java:585)
> 2006-11-20 16:58:39.560967500   at
> org.red5.server.service.ServiceInvoker.invoke  
(ServiceInvoker.java:161)
> 2006-11-20 16:58:39.560969500   at
> org.red5.server.net.rtmp.RTMPHandler.invokeCall 
(RTMPHandler.java:287)
> 2006-11-20 16:58:39.560978500   at
> org.red5.server.net.rtmp.RTMPHandler.onInvoke  
(RTMPHandler.java:469)
> 2006-11-20 16:58:39.560980500   at
> org.red5.server.net.rtmp.RTMPHandler.messageReceived 
(RTMPHandler.java:
> 147)
> 2006-11-20 16:58:39.560982500   at
> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
> (RTMPMinaIoHandler.java:78)
> 2006-11-20 16:58:39.560985500   at
> org.apache.mina.common.support.AbstractIoFilterChain 
$2.messageReceived
> (AbstractIoFilterChain.java:189)
> 2006-11-20 16:58: 39.560993500   at
>  
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessag 
eRe
> ce
> ived(AbstractIoFilterChain.java:502)
> 2006-11-20 16:58:39.560996500   at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1000
> (AbstractIoFilterChain.java:52)
> 2006-11-20 16:58:39.560998500   at
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> $1.messageReceived(AbstractIoFilterChain.java:777)
> 2006-11-20 16:58: 39.561012500   at
> org.red5.io.filter.ExecutorFilter.processEvent 
(ExecutorFilter.java:
> 231)
> 2006-11-20 16:58:39.561015500   at  
org.red5.io.filter.ExecutorFilter
> $ProcessEventsRunnable.run(ExecutorFilter.java :279)
> 2006-11-20 16:58:39.561017500   at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:650)
> 2006-11-20 16:58:39.561019500   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:675)
> 2006-11-20 16:58:39.561028500   at java.lang.Thread.run 
(Thread.java:
> 595)
>
>
> On Nov 18, 2006, at 2:48 AM, Steven Gong wrote:
>
>> Hi Paul,
>> I have just committed some codes to fix the OutOfMemoryError.  
The
>> related revision numbers are 1555, 1556 and 1557.
>>
>> The OutOfMemoryError is mainly due to the buffer in FLVReader  
so I
>> added a new static oroperty 'maxBufferSize' to constrain the  
buffer
>> size and also added a way to disable the buffer if none of  
'auto',
>> 'direct' and 'heap' buffer type is configured. The property can
>> also be configured in red5-common.xml.
>>
>> Another improvement is to let ICacheStore create ICacheable
>> instances so that in the case of NoCacheImpl, no instances  
will be
>> created thus we won't create byte array in this case.
>>
>> Could you please have a review? Thanks. :-)
>>
>> As the OutOfMemoryError bug gives rise to many complaints of our
>> users, I also cc this mail to red5 list for notification  
purpose.
>>
>> --
>> I cannot tell why this heart languishes in silence. It is for  
small
>> needs it never asks, or knows or remembers.  -- Tagore
>>
>> Best Regards
>> Steven Gong
>> _______________________________________________
>> Red5 mailing list
>> Red5@osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5@osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
Red5@osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org



--
I cannot tell why this heart languishes in silence. It is for  
small needs it never asks, or knows or remembers.  -- Tagore

Best Regards
Steven Gong
_______________________________________________
Red5 mailing list
Red5@osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5@osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5@osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5@osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5@osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org