One test that I would like to perform would be to use a pure red5 server instance and use one of the sample applications and simply stream a Video stream to it using the ScreenSharing Codec (for instance by simply pointing our Screensharing app to send to this webapp). I would guess it still hits the 100% CPU idle on the red5 process. That would proof that it has really nothing todo with the code that what we do inside of OpenMeetings. And we have an reproducible simple use-case. We could use that to go to the red5 list and to also investigate further which red5 version are suitable for us.
Sebastian 2014-03-18 10:24 GMT+13:00 seba.wag...@gmail.com <seba.wag...@gmail.com>: > "I can confirm higher server CPU usage on recording, will try to find the > reason." > Glad you could reproduce it. I have not seen this behavior in past red5 > versions. I think the reason is somewhere inside of the Red5 API in the way > it processes the video packets. Cause even if you comment out the > ScreenListeners to attach to the stream, the red5 process does still hit > the CPU. > > When switching to the latest Red5 release I could see that the CPU impact > was not so big anymore. However there was still one. > > "[INFO] [NioProcessor-4] org.red5.server.stream.codec.ScreenVideo - > Allocating memory for 748 compressed blocks. > I believe this is caused by creating CachedEvent and copy byte buffers" > I don't think so. Even when I had the StreamListeners commented out and > the CachedEvent is never used, I think I could see this event in the log. > It is an internal Red5 element. And somehow I think it has something todo > how red5 internally processes the video stream when it is incoming to make > it available as a stream where somebody can subscribe to. > However to be investigated. > > Sebastian > > > > 2014-03-17 16:43 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com>: > > I can confirm higher server CPU usage on recording, will try to find the >> reason. >> >> >> On Mon, Mar 17, 2014 at 8:54 AM, Maxim Solodovnik <solomax...@gmail.com >> >wrote: >> >> > They have moved to git less than a month ago :) >> > I was going to update our build to use EGit but had no time for this :(( >> > I'll check the EGit (or will ask it's developers) if it can clone/update >> > to the specific git revision. >> > >> > Tags will work as long as we will stay on the release :) Additionally we >> > can fork their repo and stay on the revision we need, but I would avoid >> > this if possible. >> > >> > >> > On Mon, Mar 17, 2014 at 4:31 AM, seba.wag...@gmail.com < >> > seba.wag...@gmail.com> wrote: >> > >> >> What concerns me most is currently the red5 server process cpu while >> >> recording. >> >> "5) red5 version was more or less up to date in the trunk (4756)," >> >> Trunk is doing a Git checkout, so our old system using red5 revision >> >> numbers is not applicable anymore. >> >> I think the build.xml should be changed so that it does not checkout >> HEAD >> >> of https://github.com/Red5/red5-server.git, instead it should >> checkout a >> >> tag. I will ask the red5-devs to create a tag. I wonder why they did >> not >> >> do >> >> that in case they do a release or major milestone. >> >> >> >> Sebastian >> >> >> >> >> >> >> >> >> >> >> >> 2014-03-17 0:16 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com>: >> >> >> >> > Detailed answers: >> >> > >> >> > 1) there is now a FPS per second changer. How does that incorporate >> with >> >> > the recordings produced? >> >> > Video frames are now being captured using constant delay timer task >> >> (with >> >> > the FPS based delay) >> >> > If none frames are ready to be sent, "no_change" frame is sended >> >> > >> >> > 2) Is the audio and the video in sync no matter what FPS I choose? >> >> > I believe so, Vasiliy has tested it and found no issues (if I'm not >> >> > mistaken), additional testing might be required >> >> > >> >> > 3) Did anybody monitor the red5 server process CPU while doing a >> >> > screen-sharing >> >> > recording? >> >> > No, we can perform such testing >> >> > >> >> > 4) CPU usage jumps to 100% whenever I start recording. >> >> > I can also see lots of statements similar to this in the log output: >> >> > [INFO] [NioProcessor-4] org.red5.server.stream.codec.ScreenVideo - >> >> > Allocating memory for 748 compressed blocks. >> >> > I believe this is caused by creating CachedEvent and copy byte >> buffers >> >> > >> >> > 5) red5 version was more or less up to date in the trunk (4756), >> >> currently >> >> > trunk is updated to the latest git version (need to be additionaly >> >> > tested/fixed) >> >> > a) client and server versions should be fixed (or we will have >> build >> >> > broken or unstable one day) >> >> > b) screen sharing is broken (need to be investigated/fixed) >> >> > c) up to r4756 red5 server was unstable while video is published >> this >> >> > should be tested more carefully >> >> > >> >> > 6) I think there is also a need to do it because partially some >> >> ressources >> >> > are no more available in the SVN repository >> >> > We have all necessary resources in our repocitory (just in case) >> >> > >> >> > 7) We could also consider downloading red5 server/client from Jenkins >> >> > I don't think it is good idea since we don't need HEAD version all >> the >> >> time >> >> > I'm trying to rewrite our build to be maven based (not very >> successful >> >> so >> >> > far) so I guess things will change a lot if this step will be >> >> implemented. >> >> > Until then I would leave the build as it is now >> >> > >> >> > 8) My red5 CPU load is also fine as long as I don't record something. >> >> > I'll try to double check on my machines >> >> > >> >> > >> >> > >> >> > On Sun, Mar 16, 2014 at 5:06 PM, seba.wag...@gmail.com < >> >> > seba.wag...@gmail.com> wrote: >> >> > >> >> > > My red5 CPU load is also fine as long as I don't record something. >> >> > > >> >> > > Thanks, >> >> > > Sebastian >> >> > > On 16 Mar 2014 19:38, "Maxim Solodovnik" <solomax...@gmail.com> >> >> wrote: >> >> > > >> >> > > > trunk is building red5 using maven already >> >> > > > I'll review the code and merge compilation from the trunk. >> >> > > > >> >> > > > red5 was not updated in 3.0.x branch since video calls were >> broken >> >> > (still >> >> > > > broken in trunk, will check the release) >> >> > > > >> >> > > > I have not monitored server CPU, but it seems to be ~1% on my >> >> machine >> >> > > > >> >> > > > Will double check and provide detailed answers to all of your >> >> questions >> >> > > > later today :) >> >> > > > >> >> > > > >> >> > > > On Sun, Mar 16, 2014 at 12:53 PM, seba.wag...@gmail.com < >> >> > > > seba.wag...@gmail.com> wrote: >> >> > > > >> >> > > > > Btw: We could also consider downloading red5 server/client from >> >> > > Jenkins: >> >> > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://builds.apache.org/view/M-R/view/OpenMeetings/job/Red5-server/lastSuccessfulBuild/artifact/target/red5-server-1.0.2-M1-server.zip >> >> > > > > >> >> > > > > Although integrating Git and Maven build into ANT is also >> >> possible. >> >> > > > > >> >> > > > > What is the preference ? >> >> > > > > >> >> > > > > Sebastian >> >> > > > > >> >> > > > > >> >> > > > > 2014-03-16 18:51 GMT+13:00 seba.wag...@gmail.com < >> >> > > seba.wag...@gmail.com >> >> > > > >: >> >> > > > > >> >> > > > > > http://svn.apache.org/r1577984 does fix to do a Git >> checkout >> >> > using >> >> > > > the >> >> > > > > > latest code from their repository and for the client and >> server. >> >> > > > > > The folder structure is slightly different, needs some >> further >> >> > > > > adjustments. >> >> > > > > > Also I think some of the patches (Tomcat7 patch) is no more >> >> > required >> >> > > as >> >> > > > > > the latest red5 is already using Tomcat 7. >> >> > > > > > >> >> > > > > > I have created a ticket to capture the progress: >> >> > > > > > https://issues.apache.org/jira/browse/OPENMEETINGS-950 >> >> > > > > > It will require some more work and review before this piece >> of >> >> work >> >> > > is >> >> > > > > > ready to be merged back to any of the other branches. >> >> > > > > > >> >> > > > > > However I think it might be useful for our CPU issues and >> moving >> >> > > > forward. >> >> > > > > > Getting rid of the Tomcat7 patches and SVN-kit checkout stuff >> >> seems >> >> > > to >> >> > > > > make >> >> > > > > > our life also a little bit easier. >> >> > > > > > >> >> > > > > > Sebastian >> >> > > > > > >> >> > > > > > >> >> > > > > > 2014-03-16 17:19 GMT+13:00 seba.wag...@gmail.com < >> >> > > > seba.wag...@gmail.com >> >> > > > > >: >> >> > > > > > >> >> > > > > > I think there is also a need to do it because partially some >> >> > > ressources >> >> > > > > >> are no more available in the SVN repository: >> >> > > > > >> http://red5.googlecode.com/svn/java/client/readme.txt >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> 2014-03-16 15:25 GMT+13:00 seba.wag...@gmail.com < >> >> > > > seba.wag...@gmail.com >> >> > > > > >: >> >> > > > > >> >> >> > > > > >> I also tried updating to the latest release of Red5 (1.0.2 >> >> seems >> >> > to >> >> > > be >> >> > > > > >>> just released). >> >> > > > > >>> I was more or less successful. >> >> > > > > >>> >> >> > > > > >>> When using red5 in its latest version the CPU usage when >> >> doing a >> >> > > > screen >> >> > > > > >>> sharing of the red5 server side process is a lot less. >> >> > > > > >>> >> >> > > > > >>> I can see that there are a couple of issues updating to the >> >> > latest >> >> > > > red5 >> >> > > > > >>> versions. However letting them too much out of sync was >> always >> >> > > > > difficult in >> >> > > > > >>> the past as there are regularly changes that you need to >> >> > duplicate >> >> > > in >> >> > > > > the >> >> > > > > >>> OpenMeetings API/Configuration files et cetera. And of >> course >> >> > > > > regression >> >> > > > > >>> testing is a pain. >> >> > > > > >>> However we rely on the improvements of the red5 server API. >> >> > > > > >>> >> >> > > > > >>> What is the current status of the red5 version? Our version >> >> r4393 >> >> > > is >> >> > > > > >>> from 07/2012 ( >> >> > https://code.google.com/p/red5/source/detail?r=4393) >> >> > > > :) >> >> > > > > >>> We should make a move I think. There seems to be maybe a >> good >> >> > point >> >> > > > now >> >> > > > > >>> when there is a new stable release to review a migration to >> >> the >> >> > > > latest >> >> > > > > >>> version. >> >> > > > > >>> >> >> > > > > >>> What do you think? >> >> > > > > >>> What are the current show stoppers from upgrading to the >> >> latest >> >> > > red5 >> >> > > > > >>> version? >> >> > > > > >>> I can see a couple of issues when upgrading, but it seems >> >> there >> >> > is >> >> > > no >> >> > > > > >>> major incompatibility between OpenMeetings and later Red5 >> >> > versions. >> >> > > > > Spring >> >> > > > > >>> is now 4.0 in red5. And some minor changes in the >> >> red5-web.xml. >> >> > > > > >>> And it seems like the .upload servlet is not correctly >> >> > initialized. >> >> > > > > >>> I can share my upgraded OpenMeetings instance if anybody is >> >> > > > interested. >> >> > > > > >>> >> >> > > > > >>> Sebastian >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> 2014-03-16 14:28 GMT+13:00 seba.wag...@gmail.com < >> >> > > > > seba.wag...@gmail.com> >> >> > > > > >>> : >> >> > > > > >>> >> >> > > > > >>> Regarding the Red5 CPU usage: I did a couple of tests. It >> does >> >> > not >> >> > > > seem >> >> > > > > >>>> to be like previously a writer problem (writer too slow to >> >> write >> >> > > > > packets to >> >> > > > > >>>> disk). Even if I comment out the stream listeners so that >> >> > nothing >> >> > > > > will be >> >> > > > > >>>> written to disk the CPU usage jumps to 100% whenever I >> start >> >> > > > > recording. >> >> > > > > >>>> I can also see lots of statements similar to this in the >> log >> >> > > output: >> >> > > > > >>>> [INFO] [NioProcessor-4] >> >> > org.red5.server.stream.codec.ScreenVideo - >> >> > > > > >>>> Allocating memory for 748 compressed blocks. >> >> > > > > >>>> [INFO] [NioProcessor-4] >> >> > org.red5.server.stream.codec.ScreenVideo - >> >> > > > > >>>> Allocating memory for 1305 compressed blocks. >> >> > > > > >>>> >> >> > > > > >>>> I have not seen this kind of logging output in past >> versions >> >> of >> >> > > > red5. >> >> > > > > >>>> >> >> > > > > >>>> Sebastian >> >> > > > > >>>> >> >> > > > > >>>> >> >> > > > > >>>> >> >> > > > > >>>> >> >> > > > > >>>> 2014-03-16 12:44 GMT+13:00 seba.wag...@gmail.com < >> >> > > > > seba.wag...@gmail.com >> >> > > > > >>>> >: >> >> > > > > >>>> >> >> > > > > >>>> Hi, >> >> > > > > >>>>> >> >> > > > > >>>>> there is now a FPS per second changer. How does that >> >> > incorporate >> >> > > > with >> >> > > > > >>>>> the recordings produced? Is the audio and the video in >> sync >> >> no >> >> > > > > matter what >> >> > > > > >>>>> FPS I choose? >> >> > > > > >>>>> >> >> > > > > >>>>> Did anybody monitor the red5 server process CPU while >> doing >> >> a >> >> > > > > >>>>> screen-sharing recording? I can still see the CPU jump to >> >> 100% >> >> > of >> >> > > > the >> >> > > > > >>>>> server process if I start a recording. >> >> > > > > >>>>> >> >> > > > > >>>>> It would be really good to have a demo server instead of >> >> doing >> >> > > this >> >> > > > > >>>>> local verification. >> >> > > > > >>>>> >> >> > > > > >>>>> Thanks, >> >> > > > > >>>>> Sebastian >> >> > > > > >>>>> -- >> >> > > > > >>>>> Sebastian Wagner >> >> > > > > >>>>> https://twitter.com/#!/dead_lock >> >> > > > > >>>>> http://www.webbase-design.de >> >> > > > > >>>>> http://www.wagner-sebastian.com >> >> > > > > >>>>> seba.wag...@gmail.com >> >> > > > > >>>>> >> >> > > > > >>>> >> >> > > > > >>>> >> >> > > > > >>>> >> >> > > > > >>>> -- >> >> > > > > >>>> Sebastian Wagner >> >> > > > > >>>> https://twitter.com/#!/dead_lock >> >> > > > > >>>> http://www.webbase-design.de >> >> > > > > >>>> http://www.wagner-sebastian.com >> >> > > > > >>>> seba.wag...@gmail.com >> >> > > > > >>>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> -- >> >> > > > > >>> Sebastian Wagner >> >> > > > > >>> https://twitter.com/#!/dead_lock >> >> > > > > >>> http://www.webbase-design.de >> >> > > > > >>> http://www.wagner-sebastian.com >> >> > > > > >>> seba.wag...@gmail.com >> >> > > > > >>> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> -- >> >> > > > > >> Sebastian Wagner >> >> > > > > >> https://twitter.com/#!/dead_lock >> >> > > > > >> http://www.webbase-design.de >> >> > > > > >> http://www.wagner-sebastian.com >> >> > > > > >> seba.wag...@gmail.com >> >> > > > > >> >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > -- >> >> > > > > > Sebastian Wagner >> >> > > > > > https://twitter.com/#!/dead_lock >> >> > > > > > http://www.webbase-design.de >> >> > > > > > http://www.wagner-sebastian.com >> >> > > > > > seba.wag...@gmail.com >> >> > > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > -- >> >> > > > > Sebastian Wagner >> >> > > > > https://twitter.com/#!/dead_lock >> >> > > > > http://www.webbase-design.de >> >> > > > > http://www.wagner-sebastian.com >> >> > > > > seba.wag...@gmail.com >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > -- >> >> > > > WBR >> >> > > > Maxim aka solomax >> >> > > > >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > WBR >> >> > Maxim aka solomax >> >> > >> >> >> >> >> >> >> >> -- >> >> Sebastian Wagner >> >> https://twitter.com/#!/dead_lock >> >> http://www.webbase-design.de >> >> http://www.wagner-sebastian.com >> >> seba.wag...@gmail.com >> >> >> > >> > >> > >> > -- >> > WBR >> > Maxim aka solomax >> > >> >> >> >> -- >> WBR >> Maxim aka solomax >> > > > > -- > Sebastian Wagner > https://twitter.com/#!/dead_lock > http://www.webbase-design.de > http://www.wagner-sebastian.com > seba.wag...@gmail.com > -- Sebastian Wagner https://twitter.com/#!/dead_lock http://www.webbase-design.de http://www.wagner-sebastian.com seba.wag...@gmail.com
