Will try to run with bigger picture from camera, might break everything If it will break I'll contact red5 dev
On Tue, Mar 18, 2014 at 12:34 PM, seba.wag...@gmail.com < seba.wag...@gmail.com> wrote: > " believe the higher CPU usage is caused by this line > > CachedEvent item = queue.poll(100, TimeUnit.MICROSECONDS); > I'm going to change it > CachedEvent item = queue.poll(100, TimeUnit.MILLISECONDS); > and test" > > I have commented out this Line 256 in > > https://svn.apache.org/viewvc/openmeetings/trunk/singlewebapp/src/main/java/org/apache/openmeetings/remote/FLVRecorderService.java?view=markup > > That attaches the listener to the streams ( > stream.addStreamListener(streamListener);) > > If you comment out this line none of the custom code that we do > (CachedEvent, event queue, nothing) will be ever called. As the listener is > simply not attached to the stream. > > But even then -> You will see the same impact on the CPU. So none of the > above around writing or polling has this impact on the CPU. > > > So that is why I think it has nothing to do with our implementation. > > Sebastian > > > > > 2014-03-18 18:21 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com>: > > > It seems like screen sharing is broken in case recorded area too big :( > > The time necessary for StreamPacket duplicating is too big, so the screen > > sharing recording session is being killed after series > > of java.util.concurrent.RejectedExecutionException > > > > need to handle this some how :( > > > > > > On Tue, Mar 18, 2014 at 11:08 AM, Maxim Solodovnik <solomax...@gmail.com > > >wrote: > > > > > Additionally CPU is consumed while copying incoming frames while > > CachedEvent > > > is created :( > > > I'll try to create reusable queue of buffers for it ... > > > > > > > > > On Tue, Mar 18, 2014 at 10:46 AM, Maxim Solodovnik < > solomax...@gmail.com > > >wrote: > > > > > >> I believe the higher CPU usage is caused by this line > > >> > > >> CachedEvent item = queue.poll(100, TimeUnit.MICROSECONDS); > > >> I'm going to change it > > >> CachedEvent item = queue.poll(100, TimeUnit.MILLISECONDS); > > >> and test > > >> > > >> > > >> On Tue, Mar 18, 2014 at 4:39 AM, seba.wag...@gmail.com < > > >> seba.wag...@gmail.com> wrote: > > >> > > >>> One test that I would like to perform would be to use a pure red5 > > server > > >>> instance and use one of the sample applications and simply stream a > > Video > > >>> stream to it using the ScreenSharing Codec (for instance by simply > > >>> pointing > > >>> our Screensharing app to send to this webapp). > > >>> I would guess it still hits the 100% CPU idle on the red5 process. > > >>> That would proof that it has really nothing todo with the code that > > what > > >>> we > > >>> do inside of OpenMeetings. > > >>> And we have an reproducible simple use-case. > > >>> We could use that to go to the red5 list and to also investigate > > further > > >>> which red5 version are suitable for us. > > >>> > > >>> Sebastian > > >>> > > >>> > > >>> 2014-03-18 10:24 GMT+13:00 seba.wag...@gmail.com < > > seba.wag...@gmail.com > > >>> >: > > >>> > > >>> > "I can confirm higher server CPU usage on recording, will try to > find > > >>> the > > >>> > reason." > > >>> > Glad you could reproduce it. I have not seen this behavior in past > > red5 > > >>> > versions. I think the reason is somewhere inside of the Red5 API in > > >>> the way > > >>> > it processes the video packets. Cause even if you comment out the > > >>> > ScreenListeners to attach to the stream, the red5 process does > still > > >>> hit > > >>> > the CPU. > > >>> > > > >>> > When switching to the latest Red5 release I could see that the CPU > > >>> impact > > >>> > was not so big anymore. However there was still one. > > >>> > > > >>> > "[INFO] [NioProcessor-4] org.red5.server.stream.codec.ScreenVideo - > > >>> > Allocating memory for 748 compressed blocks. > > >>> > I believe this is caused by creating CachedEvent and copy byte > > buffers" > > >>> > I don't think so. Even when I had the StreamListeners commented > out > > >>> and > > >>> > the CachedEvent is never used, I think I could see this event in > the > > >>> log. > > >>> > It is an internal Red5 element. And somehow I think it has > something > > >>> todo > > >>> > how red5 internally processes the video stream when it is incoming > to > > >>> make > > >>> > it available as a stream where somebody can subscribe to. > > >>> > However to be investigated. > > >>> > > > >>> > Sebastian > > >>> > > > >>> > > > >>> > > > >>> > 2014-03-17 16:43 GMT+13:00 Maxim Solodovnik <solomax...@gmail.com > >: > > >>> > > > >>> > I can confirm higher server CPU usage on recording, will try to > find > > >>> the > > >>> >> reason. > > >>> >> > > >>> >> > > >>> >> On Mon, Mar 17, 2014 at 8:54 AM, Maxim Solodovnik < > > >>> solomax...@gmail.com > > >>> >> >wrote: > > >>> >> > > >>> >> > They have moved to git less than a month ago :) > > >>> >> > I was going to update our build to use EGit but had no time for > > >>> this :(( > > >>> >> > I'll check the EGit (or will ask it's developers) if it can > > >>> clone/update > > >>> >> > to the specific git revision. > > >>> >> > > > >>> >> > Tags will work as long as we will stay on the release :) > > >>> Additionally we > > >>> >> > can fork their repo and stay on the revision we need, but I > would > > >>> avoid > > >>> >> > this if possible. > > >>> >> > > > >>> >> > > > >>> >> > On Mon, Mar 17, 2014 at 4:31 AM, seba.wag...@gmail.com < > > >>> >> > seba.wag...@gmail.com> wrote: > > >>> >> > > > >>> >> >> What concerns me most is currently the red5 server process cpu > > >>> while > > >>> >> >> recording. > > >>> >> >> "5) red5 version was more or less up to date in the trunk > > (4756)," > > >>> >> >> Trunk is doing a Git checkout, so our old system using red5 > > >>> revision > > >>> >> >> numbers is not applicable anymore. > > >>> >> >> I think the build.xml should be changed so that it does not > > >>> checkout > > >>> >> HEAD > > >>> >> >> of https://github.com/Red5/red5-server.git, instead it should > > >>> >> checkout a > > >>> >> >> tag. I will ask the red5-devs to create a tag. I wonder why > they > > >>> did > > >>> >> not > > >>> >> >> do > > >>> >> >> that in case they do a release or major milestone. > > >>> >> >> > > >>> >> >> Sebastian > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> 2014-03-17 0:16 GMT+13:00 Maxim Solodovnik < > solomax...@gmail.com > > >: > > >>> >> >> > > >>> >> >> > Detailed answers: > > >>> >> >> > > > >>> >> >> > 1) there is now a FPS per second changer. How does that > > >>> incorporate > > >>> >> with > > >>> >> >> > the recordings produced? > > >>> >> >> > Video frames are now being captured using constant delay > timer > > >>> task > > >>> >> >> (with > > >>> >> >> > the FPS based delay) > > >>> >> >> > If none frames are ready to be sent, "no_change" frame is > > sended > > >>> >> >> > > > >>> >> >> > 2) Is the audio and the video in sync no matter what FPS I > > >>> choose? > > >>> >> >> > I believe so, Vasiliy has tested it and found no issues (if > I'm > > >>> not > > >>> >> >> > mistaken), additional testing might be required > > >>> >> >> > > > >>> >> >> > 3) Did anybody monitor the red5 server process CPU while > doing > > a > > >>> >> >> > screen-sharing > > >>> >> >> > recording? > > >>> >> >> > No, we can perform such testing > > >>> >> >> > > > >>> >> >> > 4) CPU usage jumps to 100% whenever I start recording. > > >>> >> >> > I can also see lots of statements similar to this in the log > > >>> output: > > >>> >> >> > [INFO] [NioProcessor-4] > > org.red5.server.stream.codec.ScreenVideo > > >>> - > > >>> >> >> > Allocating memory for 748 compressed blocks. > > >>> >> >> > I believe this is caused by creating CachedEvent and copy > byte > > >>> >> buffers > > >>> >> >> > > > >>> >> >> > 5) red5 version was more or less up to date in the trunk > > (4756), > > >>> >> >> currently > > >>> >> >> > trunk is updated to the latest git version (need to be > > >>> additionaly > > >>> >> >> > tested/fixed) > > >>> >> >> > a) client and server versions should be fixed (or we will > > have > > >>> >> build > > >>> >> >> > broken or unstable one day) > > >>> >> >> > b) screen sharing is broken (need to be > investigated/fixed) > > >>> >> >> > c) up to r4756 red5 server was unstable while video is > > >>> published > > >>> >> this > > >>> >> >> > should be tested more carefully > > >>> >> >> > > > >>> >> >> > 6) I think there is also a need to do it because partially > some > > >>> >> >> ressources > > >>> >> >> > are no more available in the SVN repository > > >>> >> >> > We have all necessary resources in our repocitory (just in > > case) > > >>> >> >> > > > >>> >> >> > 7) We could also consider downloading red5 server/client from > > >>> Jenkins > > >>> >> >> > I don't think it is good idea since we don't need HEAD > version > > >>> all > > >>> >> the > > >>> >> >> time > > >>> >> >> > I'm trying to rewrite our build to be maven based (not very > > >>> >> successful > > >>> >> >> so > > >>> >> >> > far) so I guess things will change a lot if this step will be > > >>> >> >> implemented. > > >>> >> >> > Until then I would leave the build as it is now > > >>> >> >> > > > >>> >> >> > 8) My red5 CPU load is also fine as long as I don't record > > >>> something. > > >>> >> >> > I'll try to double check on my machines > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > On Sun, Mar 16, 2014 at 5:06 PM, seba.wag...@gmail.com < > > >>> >> >> > seba.wag...@gmail.com> wrote: > > >>> >> >> > > > >>> >> >> > > My red5 CPU load is also fine as long as I don't record > > >>> something. > > >>> >> >> > > > > >>> >> >> > > Thanks, > > >>> >> >> > > Sebastian > > >>> >> >> > > On 16 Mar 2014 19:38, "Maxim Solodovnik" < > > solomax...@gmail.com > > >>> > > > >>> >> >> wrote: > > >>> >> >> > > > > >>> >> >> > > > trunk is building red5 using maven already > > >>> >> >> > > > I'll review the code and merge compilation from the > trunk. > > >>> >> >> > > > > > >>> >> >> > > > red5 was not updated in 3.0.x branch since video calls > were > > >>> >> broken > > >>> >> >> > (still > > >>> >> >> > > > broken in trunk, will check the release) > > >>> >> >> > > > > > >>> >> >> > > > I have not monitored server CPU, but it seems to be ~1% > on > > my > > >>> >> >> machine > > >>> >> >> > > > > > >>> >> >> > > > Will double check and provide detailed answers to all of > > your > > >>> >> >> questions > > >>> >> >> > > > later today :) > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > On Sun, Mar 16, 2014 at 12:53 PM, seba.wag...@gmail.com< > > >>> >> >> > > > seba.wag...@gmail.com> wrote: > > >>> >> >> > > > > > >>> >> >> > > > > Btw: We could also consider downloading red5 > > server/client > > >>> from > > >>> >> >> > > Jenkins: > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> > > > https://builds.apache.org/view/M-R/view/OpenMeetings/job/Red5-server/lastSuccessfulBuild/artifact/target/red5-server-1.0.2-M1-server.zip > > >>> >> >> > > > > > > >>> >> >> > > > > Although integrating Git and Maven build into ANT is > also > > >>> >> >> possible. > > >>> >> >> > > > > > > >>> >> >> > > > > What is the preference ? > > >>> >> >> > > > > > > >>> >> >> > > > > Sebastian > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > 2014-03-16 18:51 GMT+13:00 seba.wag...@gmail.com < > > >>> >> >> > > seba.wag...@gmail.com > > >>> >> >> > > > >: > > >>> >> >> > > > > > > >>> >> >> > > > > > http://svn.apache.org/r1577984 does fix to do a Git > > >>> >> checkout > > >>> >> >> > using > > >>> >> >> > > > the > > >>> >> >> > > > > > latest code from their repository and for the client > > and > > >>> >> server. > > >>> >> >> > > > > > The folder structure is slightly different, needs > some > > >>> >> further > > >>> >> >> > > > > adjustments. > > >>> >> >> > > > > > Also I think some of the patches (Tomcat7 patch) is > no > > >>> more > > >>> >> >> > required > > >>> >> >> > > as > > >>> >> >> > > > > > the latest red5 is already using Tomcat 7. > > >>> >> >> > > > > > > > >>> >> >> > > > > > I have created a ticket to capture the progress: > > >>> >> >> > > > > > > https://issues.apache.org/jira/browse/OPENMEETINGS-950 > > >>> >> >> > > > > > It will require some more work and review before this > > >>> piece > > >>> >> of > > >>> >> >> work > > >>> >> >> > > is > > >>> >> >> > > > > > ready to be merged back to any of the other branches. > > >>> >> >> > > > > > > > >>> >> >> > > > > > However I think it might be useful for our CPU issues > > and > > >>> >> moving > > >>> >> >> > > > forward. > > >>> >> >> > > > > > Getting rid of the Tomcat7 patches and SVN-kit > checkout > > >>> stuff > > >>> >> >> seems > > >>> >> >> > > to > > >>> >> >> > > > > make > > >>> >> >> > > > > > our life also a little bit easier. > > >>> >> >> > > > > > > > >>> >> >> > > > > > Sebastian > > >>> >> >> > > > > > > > >>> >> >> > > > > > > > >>> >> >> > > > > > 2014-03-16 17:19 GMT+13:00 seba.wag...@gmail.com < > > >>> >> >> > > > seba.wag...@gmail.com > > >>> >> >> > > > > >: > > >>> >> >> > > > > > > > >>> >> >> > > > > > I think there is also a need to do it because > partially > > >>> some > > >>> >> >> > > ressources > > >>> >> >> > > > > >> are no more available in the SVN repository: > > >>> >> >> > > > > >> > http://red5.googlecode.com/svn/java/client/readme.txt > > >>> >> >> > > > > >> > > >>> >> >> > > > > >> > > >>> >> >> > > > > >> 2014-03-16 15:25 GMT+13:00 seba.wag...@gmail.com < > > >>> >> >> > > > seba.wag...@gmail.com > > >>> >> >> > > > > >: > > >>> >> >> > > > > >> > > >>> >> >> > > > > >> I also tried updating to the latest release of Red5 > > >>> (1.0.2 > > >>> >> >> seems > > >>> >> >> > to > > >>> >> >> > > be > > >>> >> >> > > > > >>> just released). > > >>> >> >> > > > > >>> I was more or less successful. > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> When using red5 in its latest version the CPU usage > > >>> when > > >>> >> >> doing a > > >>> >> >> > > > screen > > >>> >> >> > > > > >>> sharing of the red5 server side process is a lot > > less. > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> I can see that there are a couple of issues > updating > > >>> to the > > >>> >> >> > latest > > >>> >> >> > > > red5 > > >>> >> >> > > > > >>> versions. However letting them too much out of sync > > was > > >>> >> always > > >>> >> >> > > > > difficult in > > >>> >> >> > > > > >>> the past as there are regularly changes that you > need > > >>> to > > >>> >> >> > duplicate > > >>> >> >> > > in > > >>> >> >> > > > > the > > >>> >> >> > > > > >>> OpenMeetings API/Configuration files et cetera. And > > of > > >>> >> course > > >>> >> >> > > > > regression > > >>> >> >> > > > > >>> testing is a pain. > > >>> >> >> > > > > >>> However we rely on the improvements of the red5 > > server > > >>> API. > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> What is the current status of the red5 version? Our > > >>> version > > >>> >> >> r4393 > > >>> >> >> > > is > > >>> >> >> > > > > >>> from 07/2012 ( > > >>> >> >> > https://code.google.com/p/red5/source/detail?r=4393) > > >>> >> >> > > > :) > > >>> >> >> > > > > >>> We should make a move I think. There seems to be > > maybe > > >>> a > > >>> >> good > > >>> >> >> > point > > >>> >> >> > > > now > > >>> >> >> > > > > >>> when there is a new stable release to review a > > >>> migration to > > >>> >> >> the > > >>> >> >> > > > latest > > >>> >> >> > > > > >>> version. > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> What do you think? > > >>> >> >> > > > > >>> What are the current show stoppers from upgrading > to > > >>> the > > >>> >> >> latest > > >>> >> >> > > red5 > > >>> >> >> > > > > >>> version? > > >>> >> >> > > > > >>> I can see a couple of issues when upgrading, but it > > >>> seems > > >>> >> >> there > > >>> >> >> > is > > >>> >> >> > > no > > >>> >> >> > > > > >>> major incompatibility between OpenMeetings and > later > > >>> Red5 > > >>> >> >> > versions. > > >>> >> >> > > > > Spring > > >>> >> >> > > > > >>> is now 4.0 in red5. And some minor changes in the > > >>> >> >> red5-web.xml. > > >>> >> >> > > > > >>> And it seems like the .upload servlet is not > > correctly > > >>> >> >> > initialized. > > >>> >> >> > > > > >>> I can share my upgraded OpenMeetings instance if > > >>> anybody is > > >>> >> >> > > > interested. > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> Sebastian > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> 2014-03-16 14:28 GMT+13:00 seba.wag...@gmail.com < > > >>> >> >> > > > > seba.wag...@gmail.com> > > >>> >> >> > > > > >>> : > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> Regarding the Red5 CPU usage: I did a couple of > > tests. > > >>> It > > >>> >> does > > >>> >> >> > not > > >>> >> >> > > > seem > > >>> >> >> > > > > >>>> to be like previously a writer problem (writer too > > >>> slow to > > >>> >> >> write > > >>> >> >> > > > > packets to > > >>> >> >> > > > > >>>> disk). Even if I comment out the stream listeners > so > > >>> that > > >>> >> >> > nothing > > >>> >> >> > > > > will be > > >>> >> >> > > > > >>>> written to disk the CPU usage jumps to 100% > > whenever I > > >>> >> start > > >>> >> >> > > > > recording. > > >>> >> >> > > > > >>>> I can also see lots of statements similar to this > in > > >>> the > > >>> >> log > > >>> >> >> > > output: > > >>> >> >> > > > > >>>> [INFO] [NioProcessor-4] > > >>> >> >> > org.red5.server.stream.codec.ScreenVideo - > > >>> >> >> > > > > >>>> Allocating memory for 748 compressed blocks. > > >>> >> >> > > > > >>>> [INFO] [NioProcessor-4] > > >>> >> >> > org.red5.server.stream.codec.ScreenVideo - > > >>> >> >> > > > > >>>> Allocating memory for 1305 compressed blocks. > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> I have not seen this kind of logging output in > past > > >>> >> versions > > >>> >> >> of > > >>> >> >> > > > red5. > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> Sebastian > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> 2014-03-16 12:44 GMT+13:00 seba.wag...@gmail.com< > > >>> >> >> > > > > seba.wag...@gmail.com > > >>> >> >> > > > > >>>> >: > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> Hi, > > >>> >> >> > > > > >>>>> > > >>> >> >> > > > > >>>>> there is now a FPS per second changer. How does > > that > > >>> >> >> > incorporate > > >>> >> >> > > > with > > >>> >> >> > > > > >>>>> the recordings produced? Is the audio and the > video > > >>> in > > >>> >> sync > > >>> >> >> no > > >>> >> >> > > > > matter what > > >>> >> >> > > > > >>>>> FPS I choose? > > >>> >> >> > > > > >>>>> > > >>> >> >> > > > > >>>>> Did anybody monitor the red5 server process CPU > > while > > >>> >> doing > > >>> >> >> a > > >>> >> >> > > > > >>>>> screen-sharing recording? I can still see the CPU > > >>> jump to > > >>> >> >> 100% > > >>> >> >> > of > > >>> >> >> > > > the > > >>> >> >> > > > > >>>>> server process if I start a recording. > > >>> >> >> > > > > >>>>> > > >>> >> >> > > > > >>>>> It would be really good to have a demo server > > >>> instead of > > >>> >> >> doing > > >>> >> >> > > this > > >>> >> >> > > > > >>>>> local verification. > > >>> >> >> > > > > >>>>> > > >>> >> >> > > > > >>>>> Thanks, > > >>> >> >> > > > > >>>>> Sebastian > > >>> >> >> > > > > >>>>> -- > > >>> >> >> > > > > >>>>> Sebastian Wagner > > >>> >> >> > > > > >>>>> https://twitter.com/#!/dead_lock > > >>> >> >> > > > > >>>>> http://www.webbase-design.de > > >>> >> >> > > > > >>>>> http://www.wagner-sebastian.com > > >>> >> >> > > > > >>>>> seba.wag...@gmail.com > > >>> >> >> > > > > >>>>> > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>>> -- > > >>> >> >> > > > > >>>> Sebastian Wagner > > >>> >> >> > > > > >>>> https://twitter.com/#!/dead_lock > > >>> >> >> > > > > >>>> http://www.webbase-design.de > > >>> >> >> > > > > >>>> http://www.wagner-sebastian.com > > >>> >> >> > > > > >>>> seba.wag...@gmail.com > > >>> >> >> > > > > >>>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >>> -- > > >>> >> >> > > > > >>> Sebastian Wagner > > >>> >> >> > > > > >>> https://twitter.com/#!/dead_lock > > >>> >> >> > > > > >>> http://www.webbase-design.de > > >>> >> >> > > > > >>> http://www.wagner-sebastian.com > > >>> >> >> > > > > >>> seba.wag...@gmail.com > > >>> >> >> > > > > >>> > > >>> >> >> > > > > >> > > >>> >> >> > > > > >> > > >>> >> >> > > > > >> > > >>> >> >> > > > > >> -- > > >>> >> >> > > > > >> Sebastian Wagner > > >>> >> >> > > > > >> https://twitter.com/#!/dead_lock > > >>> >> >> > > > > >> http://www.webbase-design.de > > >>> >> >> > > > > >> http://www.wagner-sebastian.com > > >>> >> >> > > > > >> seba.wag...@gmail.com > > >>> >> >> > > > > >> > > >>> >> >> > > > > > > > >>> >> >> > > > > > > > >>> >> >> > > > > > > > >>> >> >> > > > > > -- > > >>> >> >> > > > > > Sebastian Wagner > > >>> >> >> > > > > > https://twitter.com/#!/dead_lock > > >>> >> >> > > > > > http://www.webbase-design.de > > >>> >> >> > > > > > http://www.wagner-sebastian.com > > >>> >> >> > > > > > seba.wag...@gmail.com > > >>> >> >> > > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > -- > > >>> >> >> > > > > Sebastian Wagner > > >>> >> >> > > > > https://twitter.com/#!/dead_lock > > >>> >> >> > > > > http://www.webbase-design.de > > >>> >> >> > > > > http://www.wagner-sebastian.com > > >>> >> >> > > > > seba.wag...@gmail.com > > >>> >> >> > > > > > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > -- > > >>> >> >> > > > WBR > > >>> >> >> > > > Maxim aka solomax > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > -- > > >>> >> >> > WBR > > >>> >> >> > Maxim aka solomax > > >>> >> >> > > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> -- > > >>> >> >> Sebastian Wagner > > >>> >> >> https://twitter.com/#!/dead_lock > > >>> >> >> http://www.webbase-design.de > > >>> >> >> http://www.wagner-sebastian.com > > >>> >> >> seba.wag...@gmail.com > > >>> >> >> > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > -- > > >>> >> > WBR > > >>> >> > Maxim aka solomax > > >>> >> > > > >>> >> > > >>> >> > > >>> >> > > >>> >> -- > > >>> >> WBR > > >>> >> Maxim aka solomax > > >>> >> > > >>> > > > >>> > > > >>> > > > >>> > -- > > >>> > Sebastian Wagner > > >>> > https://twitter.com/#!/dead_lock > > >>> > http://www.webbase-design.de > > >>> > http://www.wagner-sebastian.com > > >>> > seba.wag...@gmail.com > > >>> > > > >>> > > >>> > > >>> > > >>> -- > > >>> Sebastian Wagner > > >>> https://twitter.com/#!/dead_lock > > >>> http://www.webbase-design.de > > >>> http://www.wagner-sebastian.com > > >>> seba.wag...@gmail.com > > >>> > > >> > > >> > > >> > > >> -- > > >> WBR > > >> Maxim aka solomax > > >> > > > > > > > > > > > > -- > > > WBR > > > Maxim aka solomax > > > > > > > > > > > -- > > WBR > > Maxim aka solomax > > > > > > -- > Sebastian Wagner > https://twitter.com/#!/dead_lock > http://www.webbase-design.de > http://www.wagner-sebastian.com > seba.wag...@gmail.com > -- WBR Maxim aka solomax
