Dan Rossi wrote: > Some interesting info from the logs, here is at the point of rebooting, > would the logs be causing memory leaks ? > > [INFO] 7299657 pool-3-thread-16:( VODApplication.appJoin ) Client joined > app 1038 at time 1177748584250 > [INFO] 7844 pool-3-thread-9:( VODApplication.appJoin ) Client joined app > 0 at time 1177748620265 >
Ok there is a pattern, server reboots, hangs or gets outofmemory at client number 1038, that doesnt really explain concurrents though. > Dan Rossi wrote: > >> Looks like even when load balanced it blocks connections after time. Its >> all the exceptions im sure of it, no out of memory errors which is >> bizarre until i try and reboot the service which it logs all the jvm >> dumps because it hanged, ive kept the threadcount to 4 as setting it to >> the max it can do causes issues which was 2700. >> >> This one is fairly bizarre as it wasnt an outofmemory error as i usually >> get, we load files from a network drive using a UNC name like so >> file:////computername/sharename >> >> jvm 2 | 2007/04/28 00:45:32 | java.io.IOException: The specified >> network name is no longer available >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> sun.nio.ch.FileDispatcher.read0(Native Method) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> sun.nio.ch.FileDispatcher.read(Unknown Source) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> sun.nio.ch.IOUtil.read(Unknown Source) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> sun.nio.ch.FileChannelImpl.read(Unknown Source) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> org.red5.io.flv.impl.FLVReader.fillBuffer(FLVReader.java:319) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> org.red5.io.flv.impl.FLVReader.fillBuffer(FLVReader.java:268) >> INFO | jvm 2 | 2007/04/28 00:45:32 | at >> org.red5.io.flv.impl.FLVReader.readTag(FLVReader.java:545) >> >> Dan Rossi wrote: >> >> >>> Seemed to kind of stablise for a while, however im looking at the >>> performance monitor and the page faults column is high, is this an issue >>> ? It reached about 1900 threads and then rebooted from the outofmemory >>> errors. I set the threadcount to 2700 as that is aparantly the max the >>> JVM can do. >>> >>> Dan Rossi wrote: >>> >>> >>> >>>> I tried adding these into the wrapper config to no avail, causes to use >>>> more virtual memory , and memory gets quite large >>>> >>>> wrapper.java.additional.1=-Xms1024m >>>> wrapper.java.additional.2=-Xmx1024m >>>> #wrapper.java.additional.3=-XX:NewSize=512m >>>> #wrapper.java.additional.4=-XX:MaxNewSize=512m >>>> #wrapper.java.additional.5=-XX:SurvivorRatio=8 >>>> >>>> Dan Rossi wrote: >>>> >>>> >>>> >>>> >>>>> Also is there any reason why we need so much memory resources, our >>>>> traffic has always been the same, and windows media only uses up about >>>>> 500MB. will we need to get 4GB now ? >>>>> >>>>> Dan Rossi wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> It seems after messing with the memory limits, red5 rejects connections >>>>>> but doesnt throw exceptions or out of memory errors, and cant connect >>>>>> until it reboots. Is there any recommendations of tweaking a windows >>>>>> server with 2GB of ram, ie tweaking the memory and cpu allocation and >>>>>> pagefile size etc. what other java options should i be adding ? How can >>>>>> i monitor what red5 is doing internally ? >>>>>> >>>>>> >>>>>> Dan Rossi wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Nothing special, its being caused every 2 hours. >>>>>>> >>>>>>> Main server is just running red5, the 2 slave machines are also running >>>>>>> windows media and IIS which were also getting the errors however the >>>>>>> server has 2GB of ram and the resource monitor only shows 700MB being >>>>>>> used etc. Its only started happening since updates not long ago. >>>>>>> >>>>>>> Lenny Sorey wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Dan, >>>>>>>> >>>>>>>> Forgive me if you have answered these questions previously. >>>>>>>> >>>>>>>> What are you running in RED5 which causes the out of memory problem. >>>>>>>> Video Streaming, Live Broadcast a combination of both? >>>>>>>> >>>>>>>> What all do you have actually running on these servers. >>>>>>>> >>>>>>>> RED5 (jetty), IIS, SQL Server, etc? >>>>>>>> >>>>>>>> How many users are connected when this error occurs? >>>>>>>> >>>>>>>> Can you tell in the scheme of things when and where the out of memory >>>>>>>> problems are occurring? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Lenny >>>>>>>> >>>>>>>> How long does RED5 until you receive your Out of Memory error message >>>>>>>> in the logs. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 4/26/07, *Dan Rossi* <[EMAIL PROTECTED] >>>>>>>> <mailto:[EMAIL PROTECTED]>> wrote: >>>>>>>> >>>>>>>> Hi we are constantly getting these errors on our windows 2003 >>>>>>>> servers >>>>>>>> after the latest svn updates. Im trying to work out how to tweak >>>>>>>> the >>>>>>>> resources on our windows servers to fix these kind of errors >>>>>>>> causing the >>>>>>>> java wrapper service to reboot. Im trying to work out what are the >>>>>>>> optimal memory and cpu allocation settings to use and java options >>>>>>>> for >>>>>>>> resources, we have 2G of ram, and dual P4 3G on all machines, even >>>>>>>> after >>>>>>>> load balancing its still doing this. Has only just started >>>>>>>> happening. >>>>>>>> >>>>>>>> # java.lang.OutOfMemoryError: requested 32756 bytes for >>>>>>>> ChunkPool::allocate. Out of swap space? >>>>>>>> >>>>>>>> Here is the advanced log >>>>>>>> >>>>>>>> 0x48127800 JavaThread "pool-2-thread-584" [_thread_blocked, >>>>>>>> id=8832] >>>>>>>> 0x4741e400 JavaThread "pool-2-thread-583" [_thread_blocked, >>>>>>>> id=2692] >>>>>>>> 0x51bc4400 JavaThread "pool-2-thread-582" [_thread_blocked, >>>>>>>> id=3668] >>>>>>>> 0x5325f400 JavaThread "pool-2-thread-581" [_thread_blocked, >>>>>>>> id=8752] >>>>>>>> 0x47ba8800 JavaThread "pool-2-thread-580" [_thread_blocked, >>>>>>>> id=9936] >>>>>>>> 0x50fac400 JavaThread "pool-2-thread-579" [_thread_blocked, >>>>>>>> id=10100] >>>>>>>> 0x47ec1c00 JavaThread "pool-2-thread-578" [_thread_blocked, >>>>>>>> id=7584] >>>>>>>> >>>>>>>> >>>>>>>> VM Arguments: >>>>>>>> jvm_args: -Xms512m -Xmx1000m -Djava.library.path=../lib >>>>>>>> -Dwrapper.key=z6ob3QwdVVUuY_JG -Dwrapper.port=32000 >>>>>>>> -Dwrapper.use_system_time=TRUE -Dwrapper.version=3.1.2 >>>>>>>> -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE >>>>>>>> -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 >>>>>>>> java_command: org.tanukisoftware.wrapper.WrapperSimpleApp >>>>>>>> org.red5.server.Standalone >>>>>>>> Launcher Type: SUN_STANDARD >>>>>>>> >>>>>>>> --------------- S Y S T E M --------------- >>>>>>>> >>>>>>>> OS: Windows Server 2003 family Build 3790 Service Pack 1 >>>>>>>> >>>>>>>> CPU:total 2 family 15, cmov, cx8, fxsr, mmx, sse, sse2, ht >>>>>>>> >>>>>>>> Memory: 4k page, physical 2030356k(997988k free), swap >>>>>>>> 3359568k(2099096k >>>>>>>> free) >>>>>>>> >>>>>>>> vm_info: Java HotSpot(TM) Client VM (1.6.0_01-b06) for windows-x86, >>>>>>>> built on Mar 14 2007 00:24:02 by "java_re" with unknown MS >>>>>>>> VC++:1310 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Red5 mailing list >>>>>>>> [email protected] <mailto:[email protected]> >>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>>>> <http://osflash.org/mailman/listinfo/red5_osflash.org> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Red5 mailing list >>>>>>>> [email protected] >>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Red5 mailing list >>>>>>> [email protected] >>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Red5 mailing list >>>>>> [email protected] >>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Red5 mailing list >>>>> [email protected] >>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Red5 mailing list >>>> [email protected] >>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>> >>>> >>>> >>>> >>>> >>> _______________________________________________ >>> Red5 mailing list >>> [email protected] >>> http://osflash.org/mailman/listinfo/red5_osflash.org >>> >>> >>> >>> >> _______________________________________________ >> Red5 mailing list >> [email protected] >> http://osflash.org/mailman/listinfo/red5_osflash.org >> >> >> > > > _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
