" believe the higher CPU usage is caused by this line CachedEvent item = queue.poll(100, TimeUnit.MICROSECONDS); I'm going to change it CachedEvent item = queue.poll(100, TimeUnit.MILLISECONDS); and test"
I have commented out this Line 256 in https://svn.apache.org/viewvc/openmeetings/trunk/singlewebapp/src/main/java/org/apache/openmeetings/remote/FLVRecorderService.java?view=markup That attaches the listener to the streams ( stream.addStreamListener(streamListener);) If you comment out this line none of the custom code that we do (CachedEvent, event queue, nothing) will be ever called. As the listener is simply not attached to the stream. But even then -> You will see the same impact on the CPU. So none of the above around writing or polling has this impact on the CPU. So that is why I think it has nothing to do with our implementation. Sebastian 2014-03-18 18:21 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com>: > It seems like screen sharing is broken in case recorded area too big :( > The time necessary for StreamPacket duplicating is too big, so the screen > sharing recording session is being killed after series > of java.util.concurrent.RejectedExecutionException > > need to handle this some how :( > > > On Tue, Mar 18, 2014 at 11:08 AM, Maxim Solodovnik <solomax...@gmail.com > >wrote: > > > Additionally CPU is consumed while copying incoming frames while > CachedEvent > > is created :( > > I'll try to create reusable queue of buffers for it ... > > > > > > On Tue, Mar 18, 2014 at 10:46 AM, Maxim Solodovnik <solomax...@gmail.com > >wrote: > > > >> I believe the higher CPU usage is caused by this line > >> > >> CachedEvent item = queue.poll(100, TimeUnit.MICROSECONDS); > >> I'm going to change it > >> CachedEvent item = queue.poll(100, TimeUnit.MILLISECONDS); > >> and test > >> > >> > >> On Tue, Mar 18, 2014 at 4:39 AM, seba.wag...@gmail.com < > >> seba.wag...@gmail.com> wrote: > >> > >>> One test that I would like to perform would be to use a pure red5 > server > >>> instance and use one of the sample applications and simply stream a > Video > >>> stream to it using the ScreenSharing Codec (for instance by simply > >>> pointing > >>> our Screensharing app to send to this webapp). > >>> I would guess it still hits the 100% CPU idle on the red5 process. > >>> That would proof that it has really nothing todo with the code that > what > >>> we > >>> do inside of OpenMeetings. > >>> And we have an reproducible simple use-case. > >>> We could use that to go to the red5 list and to also investigate > further > >>> which red5 version are suitable for us. > >>> > >>> Sebastian > >>> > >>> > >>> 2014-03-18 10:24 GMT+13:00 seba.wag...@gmail.com < > seba.wag...@gmail.com > >>> >: > >>> > >>> > "I can confirm higher server CPU usage on recording, will try to find > >>> the > >>> > reason." > >>> > Glad you could reproduce it. I have not seen this behavior in past > red5 > >>> > versions. I think the reason is somewhere inside of the Red5 API in > >>> the way > >>> > it processes the video packets. Cause even if you comment out the > >>> > ScreenListeners to attach to the stream, the red5 process does still > >>> hit > >>> > the CPU. > >>> > > >>> > When switching to the latest Red5 release I could see that the CPU > >>> impact > >>> > was not so big anymore. However there was still one. > >>> > > >>> > "[INFO] [NioProcessor-4] org.red5.server.stream.codec.ScreenVideo - > >>> > Allocating memory for 748 compressed blocks. > >>> > I believe this is caused by creating CachedEvent and copy byte > buffers" > >>> > I don't think so. Even when I had the StreamListeners commented out > >>> and > >>> > the CachedEvent is never used, I think I could see this event in the > >>> log. > >>> > It is an internal Red5 element. And somehow I think it has something > >>> todo > >>> > how red5 internally processes the video stream when it is incoming to > >>> make > >>> > it available as a stream where somebody can subscribe to. > >>> > However to be investigated. > >>> > > >>> > Sebastian > >>> > > >>> > > >>> > > >>> > 2014-03-17 16:43 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com>: > >>> > > >>> > I can confirm higher server CPU usage on recording, will try to find > >>> the > >>> >> reason. > >>> >> > >>> >> > >>> >> On Mon, Mar 17, 2014 at 8:54 AM, Maxim Solodovnik < > >>> solomax...@gmail.com > >>> >> >wrote: > >>> >> > >>> >> > They have moved to git less than a month ago :) > >>> >> > I was going to update our build to use EGit but had no time for > >>> this :(( > >>> >> > I'll check the EGit (or will ask it's developers) if it can > >>> clone/update > >>> >> > to the specific git revision. > >>> >> > > >>> >> > Tags will work as long as we will stay on the release :) > >>> Additionally we > >>> >> > can fork their repo and stay on the revision we need, but I would > >>> avoid > >>> >> > this if possible. > >>> >> > > >>> >> > > >>> >> > On Mon, Mar 17, 2014 at 4:31 AM, seba.wag...@gmail.com < > >>> >> > seba.wag...@gmail.com> wrote: > >>> >> > > >>> >> >> What concerns me most is currently the red5 server process cpu > >>> while > >>> >> >> recording. > >>> >> >> "5) red5 version was more or less up to date in the trunk > (4756)," > >>> >> >> Trunk is doing a Git checkout, so our old system using red5 > >>> revision > >>> >> >> numbers is not applicable anymore. > >>> >> >> I think the build.xml should be changed so that it does not > >>> checkout > >>> >> HEAD > >>> >> >> of https://github.com/Red5/red5-server.git, instead it should > >>> >> checkout a > >>> >> >> tag. I will ask the red5-devs to create a tag. I wonder why they > >>> did > >>> >> not > >>> >> >> do > >>> >> >> that in case they do a release or major milestone. > >>> >> >> > >>> >> >> Sebastian > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> 2014-03-17 0:16 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com > >: > >>> >> >> > >>> >> >> > Detailed answers: > >>> >> >> > > >>> >> >> > 1) there is now a FPS per second changer. How does that > >>> incorporate > >>> >> with > >>> >> >> > the recordings produced? > >>> >> >> > Video frames are now being captured using constant delay timer > >>> task > >>> >> >> (with > >>> >> >> > the FPS based delay) > >>> >> >> > If none frames are ready to be sent, "no_change" frame is > sended > >>> >> >> > > >>> >> >> > 2) Is the audio and the video in sync no matter what FPS I > >>> choose? > >>> >> >> > I believe so, Vasiliy has tested it and found no issues (if I'm > >>> not > >>> >> >> > mistaken), additional testing might be required > >>> >> >> > > >>> >> >> > 3) Did anybody monitor the red5 server process CPU while doing > a > >>> >> >> > screen-sharing > >>> >> >> > recording? > >>> >> >> > No, we can perform such testing > >>> >> >> > > >>> >> >> > 4) CPU usage jumps to 100% whenever I start recording. > >>> >> >> > I can also see lots of statements similar to this in the log > >>> output: > >>> >> >> > [INFO] [NioProcessor-4] > org.red5.server.stream.codec.ScreenVideo > >>> - > >>> >> >> > Allocating memory for 748 compressed blocks. > >>> >> >> > I believe this is caused by creating CachedEvent and copy byte > >>> >> buffers > >>> >> >> > > >>> >> >> > 5) red5 version was more or less up to date in the trunk > (4756), > >>> >> >> currently > >>> >> >> > trunk is updated to the latest git version (need to be > >>> additionaly > >>> >> >> > tested/fixed) > >>> >> >> > a) client and server versions should be fixed (or we will > have > >>> >> build > >>> >> >> > broken or unstable one day) > >>> >> >> > b) screen sharing is broken (need to be investigated/fixed) > >>> >> >> > c) up to r4756 red5 server was unstable while video is > >>> published > >>> >> this > >>> >> >> > should be tested more carefully > >>> >> >> > > >>> >> >> > 6) I think there is also a need to do it because partially some > >>> >> >> ressources > >>> >> >> > are no more available in the SVN repository > >>> >> >> > We have all necessary resources in our repocitory (just in > case) > >>> >> >> > > >>> >> >> > 7) We could also consider downloading red5 server/client from > >>> Jenkins > >>> >> >> > I don't think it is good idea since we don't need HEAD version > >>> all > >>> >> the > >>> >> >> time > >>> >> >> > I'm trying to rewrite our build to be maven based (not very > >>> >> successful > >>> >> >> so > >>> >> >> > far) so I guess things will change a lot if this step will be > >>> >> >> implemented. > >>> >> >> > Until then I would leave the build as it is now > >>> >> >> > > >>> >> >> > 8) My red5 CPU load is also fine as long as I don't record > >>> something. > >>> >> >> > I'll try to double check on my machines > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > On Sun, Mar 16, 2014 at 5:06 PM, seba.wag...@gmail.com < > >>> >> >> > seba.wag...@gmail.com> wrote: > >>> >> >> > > >>> >> >> > > My red5 CPU load is also fine as long as I don't record > >>> something. > >>> >> >> > > > >>> >> >> > > Thanks, > >>> >> >> > > Sebastian > >>> >> >> > > On 16 Mar 2014 19:38, "Maxim Solodovnik" < > solomax...@gmail.com > >>> > > >>> >> >> wrote: > >>> >> >> > > > >>> >> >> > > > trunk is building red5 using maven already > >>> >> >> > > > I'll review the code and merge compilation from the trunk. > >>> >> >> > > > > >>> >> >> > > > red5 was not updated in 3.0.x branch since video calls were > >>> >> broken > >>> >> >> > (still > >>> >> >> > > > broken in trunk, will check the release) > >>> >> >> > > > > >>> >> >> > > > I have not monitored server CPU, but it seems to be ~1% on > my > >>> >> >> machine > >>> >> >> > > > > >>> >> >> > > > Will double check and provide detailed answers to all of > your > >>> >> >> questions > >>> >> >> > > > later today :) > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > On Sun, Mar 16, 2014 at 12:53 PM, seba.wag...@gmail.com < > >>> >> >> > > > seba.wag...@gmail.com> wrote: > >>> >> >> > > > > >>> >> >> > > > > Btw: We could also consider downloading red5 > server/client > >>> from > >>> >> >> > > Jenkins: > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> > https://builds.apache.org/view/M-R/view/OpenMeetings/job/Red5-server/lastSuccessfulBuild/artifact/target/red5-server-1.0.2-M1-server.zip > >>> >> >> > > > > > >>> >> >> > > > > Although integrating Git and Maven build into ANT is also > >>> >> >> possible. > >>> >> >> > > > > > >>> >> >> > > > > What is the preference ? > >>> >> >> > > > > > >>> >> >> > > > > Sebastian > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > 2014-03-16 18:51 GMT+13:00 seba.wag...@gmail.com < > >>> >> >> > > seba.wag...@gmail.com > >>> >> >> > > > >: > >>> >> >> > > > > > >>> >> >> > > > > > http://svn.apache.org/r1577984 does fix to do a Git > >>> >> checkout > >>> >> >> > using > >>> >> >> > > > the > >>> >> >> > > > > > latest code from their repository and for the client > and > >>> >> server. > >>> >> >> > > > > > The folder structure is slightly different, needs some > >>> >> further > >>> >> >> > > > > adjustments. > >>> >> >> > > > > > Also I think some of the patches (Tomcat7 patch) is no > >>> more > >>> >> >> > required > >>> >> >> > > as > >>> >> >> > > > > > the latest red5 is already using Tomcat 7. > >>> >> >> > > > > > > >>> >> >> > > > > > I have created a ticket to capture the progress: > >>> >> >> > > > > > https://issues.apache.org/jira/browse/OPENMEETINGS-950 > >>> >> >> > > > > > It will require some more work and review before this > >>> piece > >>> >> of > >>> >> >> work > >>> >> >> > > is > >>> >> >> > > > > > ready to be merged back to any of the other branches. > >>> >> >> > > > > > > >>> >> >> > > > > > However I think it might be useful for our CPU issues > and > >>> >> moving > >>> >> >> > > > forward. > >>> >> >> > > > > > Getting rid of the Tomcat7 patches and SVN-kit checkout > >>> stuff > >>> >> >> seems > >>> >> >> > > to > >>> >> >> > > > > make > >>> >> >> > > > > > our life also a little bit easier. > >>> >> >> > > > > > > >>> >> >> > > > > > Sebastian > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > 2014-03-16 17:19 GMT+13:00 seba.wag...@gmail.com < > >>> >> >> > > > seba.wag...@gmail.com > >>> >> >> > > > > >: > >>> >> >> > > > > > > >>> >> >> > > > > > I think there is also a need to do it because partially > >>> some > >>> >> >> > > ressources > >>> >> >> > > > > >> are no more available in the SVN repository: > >>> >> >> > > > > >> http://red5.googlecode.com/svn/java/client/readme.txt > >>> >> >> > > > > >> > >>> >> >> > > > > >> > >>> >> >> > > > > >> 2014-03-16 15:25 GMT+13:00 seba.wag...@gmail.com < > >>> >> >> > > > seba.wag...@gmail.com > >>> >> >> > > > > >: > >>> >> >> > > > > >> > >>> >> >> > > > > >> I also tried updating to the latest release of Red5 > >>> (1.0.2 > >>> >> >> seems > >>> >> >> > to > >>> >> >> > > be > >>> >> >> > > > > >>> just released). > >>> >> >> > > > > >>> I was more or less successful. > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> When using red5 in its latest version the CPU usage > >>> when > >>> >> >> doing a > >>> >> >> > > > screen > >>> >> >> > > > > >>> sharing of the red5 server side process is a lot > less. > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> I can see that there are a couple of issues updating > >>> to the > >>> >> >> > latest > >>> >> >> > > > red5 > >>> >> >> > > > > >>> versions. However letting them too much out of sync > was > >>> >> always > >>> >> >> > > > > difficult in > >>> >> >> > > > > >>> the past as there are regularly changes that you need > >>> to > >>> >> >> > duplicate > >>> >> >> > > in > >>> >> >> > > > > the > >>> >> >> > > > > >>> OpenMeetings API/Configuration files et cetera. And > of > >>> >> course > >>> >> >> > > > > regression > >>> >> >> > > > > >>> testing is a pain. > >>> >> >> > > > > >>> However we rely on the improvements of the red5 > server > >>> API. > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> What is the current status of the red5 version? Our > >>> version > >>> >> >> r4393 > >>> >> >> > > is > >>> >> >> > > > > >>> from 07/2012 ( > >>> >> >> > https://code.google.com/p/red5/source/detail?r=4393) > >>> >> >> > > > :) > >>> >> >> > > > > >>> We should make a move I think. There seems to be > maybe > >>> a > >>> >> good > >>> >> >> > point > >>> >> >> > > > now > >>> >> >> > > > > >>> when there is a new stable release to review a > >>> migration to > >>> >> >> the > >>> >> >> > > > latest > >>> >> >> > > > > >>> version. > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> What do you think? > >>> >> >> > > > > >>> What are the current show stoppers from upgrading to > >>> the > >>> >> >> latest > >>> >> >> > > red5 > >>> >> >> > > > > >>> version? > >>> >> >> > > > > >>> I can see a couple of issues when upgrading, but it > >>> seems > >>> >> >> there > >>> >> >> > is > >>> >> >> > > no > >>> >> >> > > > > >>> major incompatibility between OpenMeetings and later > >>> Red5 > >>> >> >> > versions. > >>> >> >> > > > > Spring > >>> >> >> > > > > >>> is now 4.0 in red5. And some minor changes in the > >>> >> >> red5-web.xml. > >>> >> >> > > > > >>> And it seems like the .upload servlet is not > correctly > >>> >> >> > initialized. > >>> >> >> > > > > >>> I can share my upgraded OpenMeetings instance if > >>> anybody is > >>> >> >> > > > interested. > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> Sebastian > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> 2014-03-16 14:28 GMT+13:00 seba.wag...@gmail.com < > >>> >> >> > > > > seba.wag...@gmail.com> > >>> >> >> > > > > >>> : > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> Regarding the Red5 CPU usage: I did a couple of > tests. > >>> It > >>> >> does > >>> >> >> > not > >>> >> >> > > > seem > >>> >> >> > > > > >>>> to be like previously a writer problem (writer too > >>> slow to > >>> >> >> write > >>> >> >> > > > > packets to > >>> >> >> > > > > >>>> disk). Even if I comment out the stream listeners so > >>> that > >>> >> >> > nothing > >>> >> >> > > > > will be > >>> >> >> > > > > >>>> written to disk the CPU usage jumps to 100% > whenever I > >>> >> start > >>> >> >> > > > > recording. > >>> >> >> > > > > >>>> I can also see lots of statements similar to this in > >>> the > >>> >> log > >>> >> >> > > output: > >>> >> >> > > > > >>>> [INFO] [NioProcessor-4] > >>> >> >> > org.red5.server.stream.codec.ScreenVideo - > >>> >> >> > > > > >>>> Allocating memory for 748 compressed blocks. > >>> >> >> > > > > >>>> [INFO] [NioProcessor-4] > >>> >> >> > org.red5.server.stream.codec.ScreenVideo - > >>> >> >> > > > > >>>> Allocating memory for 1305 compressed blocks. > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> I have not seen this kind of logging output in past > >>> >> versions > >>> >> >> of > >>> >> >> > > > red5. > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> Sebastian > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> 2014-03-16 12:44 GMT+13:00 seba.wag...@gmail.com < > >>> >> >> > > > > seba.wag...@gmail.com > >>> >> >> > > > > >>>> >: > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> Hi, > >>> >> >> > > > > >>>>> > >>> >> >> > > > > >>>>> there is now a FPS per second changer. How does > that > >>> >> >> > incorporate > >>> >> >> > > > with > >>> >> >> > > > > >>>>> the recordings produced? Is the audio and the video > >>> in > >>> >> sync > >>> >> >> no > >>> >> >> > > > > matter what > >>> >> >> > > > > >>>>> FPS I choose? > >>> >> >> > > > > >>>>> > >>> >> >> > > > > >>>>> Did anybody monitor the red5 server process CPU > while > >>> >> doing > >>> >> >> a > >>> >> >> > > > > >>>>> screen-sharing recording? I can still see the CPU > >>> jump to > >>> >> >> 100% > >>> >> >> > of > >>> >> >> > > > the > >>> >> >> > > > > >>>>> server process if I start a recording. > >>> >> >> > > > > >>>>> > >>> >> >> > > > > >>>>> It would be really good to have a demo server > >>> instead of > >>> >> >> doing > >>> >> >> > > this > >>> >> >> > > > > >>>>> local verification. > >>> >> >> > > > > >>>>> > >>> >> >> > > > > >>>>> Thanks, > >>> >> >> > > > > >>>>> Sebastian > >>> >> >> > > > > >>>>> -- > >>> >> >> > > > > >>>>> Sebastian Wagner > >>> >> >> > > > > >>>>> https://twitter.com/#!/dead_lock > >>> >> >> > > > > >>>>> http://www.webbase-design.de > >>> >> >> > > > > >>>>> http://www.wagner-sebastian.com > >>> >> >> > > > > >>>>> seba.wag...@gmail.com > >>> >> >> > > > > >>>>> > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>>> -- > >>> >> >> > > > > >>>> Sebastian Wagner > >>> >> >> > > > > >>>> https://twitter.com/#!/dead_lock > >>> >> >> > > > > >>>> http://www.webbase-design.de > >>> >> >> > > > > >>>> http://www.wagner-sebastian.com > >>> >> >> > > > > >>>> seba.wag...@gmail.com > >>> >> >> > > > > >>>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> > >>> >> >> > > > > >>> -- > >>> >> >> > > > > >>> Sebastian Wagner > >>> >> >> > > > > >>> https://twitter.com/#!/dead_lock > >>> >> >> > > > > >>> http://www.webbase-design.de > >>> >> >> > > > > >>> http://www.wagner-sebastian.com > >>> >> >> > > > > >>> seba.wag...@gmail.com > >>> >> >> > > > > >>> > >>> >> >> > > > > >> > >>> >> >> > > > > >> > >>> >> >> > > > > >> > >>> >> >> > > > > >> -- > >>> >> >> > > > > >> Sebastian Wagner > >>> >> >> > > > > >> https://twitter.com/#!/dead_lock > >>> >> >> > > > > >> http://www.webbase-design.de > >>> >> >> > > > > >> http://www.wagner-sebastian.com > >>> >> >> > > > > >> seba.wag...@gmail.com > >>> >> >> > > > > >> > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > -- > >>> >> >> > > > > > Sebastian Wagner > >>> >> >> > > > > > https://twitter.com/#!/dead_lock > >>> >> >> > > > > > http://www.webbase-design.de > >>> >> >> > > > > > http://www.wagner-sebastian.com > >>> >> >> > > > > > seba.wag...@gmail.com > >>> >> >> > > > > > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > -- > >>> >> >> > > > > Sebastian Wagner > >>> >> >> > > > > https://twitter.com/#!/dead_lock > >>> >> >> > > > > http://www.webbase-design.de > >>> >> >> > > > > http://www.wagner-sebastian.com > >>> >> >> > > > > seba.wag...@gmail.com > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > -- > >>> >> >> > > > WBR > >>> >> >> > > > Maxim aka solomax > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > -- > >>> >> >> > WBR > >>> >> >> > Maxim aka solomax > >>> >> >> > > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Sebastian Wagner > >>> >> >> https://twitter.com/#!/dead_lock > >>> >> >> http://www.webbase-design.de > >>> >> >> http://www.wagner-sebastian.com > >>> >> >> seba.wag...@gmail.com > >>> >> >> > >>> >> > > >>> >> > > >>> >> > > >>> >> > -- > >>> >> > WBR > >>> >> > Maxim aka solomax > >>> >> > > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> WBR > >>> >> Maxim aka solomax > >>> >> > >>> > > >>> > > >>> > > >>> > -- > >>> > Sebastian Wagner > >>> > https://twitter.com/#!/dead_lock > >>> > http://www.webbase-design.de > >>> > http://www.wagner-sebastian.com > >>> > seba.wag...@gmail.com > >>> > > >>> > >>> > >>> > >>> -- > >>> Sebastian Wagner > >>> https://twitter.com/#!/dead_lock > >>> http://www.webbase-design.de > >>> http://www.wagner-sebastian.com > >>> seba.wag...@gmail.com > >>> > >> > >> > >> > >> -- > >> WBR > >> Maxim aka solomax > >> > > > > > > > > -- > > WBR > > Maxim aka solomax > > > > > > -- > WBR > Maxim aka solomax > -- Sebastian Wagner https://twitter.com/#!/dead_lock http://www.webbase-design.de http://www.wagner-sebastian.com seba.wag...@gmail.com
