True. But because the xpath engine builds the whole message we end up buffeting the whole thing in memory.
Sent from my iPhone On Sep 14, 2012, at 11:18 PM, Andreas Veithen <andreas.veit...@gmail.com> wrote: > On Wed, Sep 12, 2012 at 10:36 PM, Hiranya Jayathilaka > <hiranya...@gmail.com> wrote: >> Hi Amila, >> >> AFAIU this improvement gets rid of the SOAP serialization overhead (at the >> cost of more memory). This is certainly an interesting idea but goes against >> the philosophy that we have been following up to now - we don't buffer any >> message content in-memory. This philosophy helps us to keep the memory usage >> at a reasonably constant level even under largest volumes of traffic. In >> general we can say the memory usage will not be highly affected by a burst >> of very large (say > 10M) messages. But the moment we start buffering >> message content in memory we loose that. > > That argument is not entirely correct. What Amila proposes is to keep > in memory the part of the message that has already been read (plus of > course a few KB read in advance). That would only add to the memory > already consumed by the Axiom object model constructed for that part > of the message. There would thus be an increase in memory usage, but > it the increase would be roughly proportional. > >> I think the proper way to improve current performance level in Synapse is to >> get Synapse to use the pass through transport by default. We already >> discussed this on the list [2] a few months back and the idea was to use the >> pass through transport by default and fix the transport so that it can also >> handle non-pass through scenarios (CBR, transform, security etc). I've >> actually done a POC for this and got pretty much all the Synapse samples to >> work except for RM scenarios. I'm currently doing some tweaks to this >> implementation and when completed I'll commit the code in. >> >> I agree with your idea about making mediators say whether they want to >> access the message content or not. Basically I had to implement this as a >> part of the above mentioned POC. However I did it by adding a new method to >> the Mediator interface (isContentAware). Each mediator returns true/false >> depending on what they do and how they are configured. For an example <log/> >> will return false where as <log level="full"/> will return true. <xslt/> >> will always return true. <filter/> will return true/false depending on the >> type of XPath expression used. This way we tackle the problem in a more >> elegant and flexible manner without involving the user. I think using >> properties for this purpose is a hack and will be a hassle to the users. >> >> With the above implementation applied to your proxy configuration, message >> will be built by the SOAPBuilder when it hits the filter mediator (due to >> the XPath). On the out sequence, neither the builder nor the formatter will >> get engaged, and the message will be simply passed through Synapse at the >> transport level. >> >> I also like your idea about implementing a more efficient XPath engine that >> only builds a part of the message. That's something we have lacked for a >> while >> >> Thanks, >> Hiranya >> >> [2] - >> http://old.nabble.com/Message-Relay-Support-for-Synapse-to33693068.html >> >> >> On Wed, Sep 12, 2012 at 9:01 AM, Amila Suriarachchi >> <amilasuriarach...@gmail.com> wrote: >>> >>> hi all, >>> >>> For SOAP message based content based routing, currently we use SOAPBuilder >>> and SOAPMessageFormater to build the soap envelop and serialize it. But >>> however in the content based routing the only message part required to read >>> is the parts that has the content need for routing. On the other hand there >>> is no need to build the response xml message if no mediation required for >>> response. >>> >>> Therefore we can make this perform better with the following steps. First >>> at the message builder level we create a buffered input stream to buffer the >>> read message and set this as a message context property. Then at the synapse >>> engine level we can copy this bufferedInput stream to out message context. >>> Now at the out message context we can access the buffered input stream and >>> use that to directly serialize the message from the input stream after >>> resetting the input stream. >>> >>> I have created the related patch here[1]. With this patch I could improve >>> the TPS for the following proxy services with 10k message from 1,700.7 to >>> 12,135.48 (25% gain). >>> >>> >>> <proxy name="CBRProxy" transports="https http" startOnLoad="true"> >>> >>> <target> >>> >>> <inSequence> >>> >>> <property name="passThroughProxy" value="true" scope="default" >>> type="STRING"/> >>> >>> <filter xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; >>> source="//order[1]/symbol" regex="IBM"> >>> >>> <then> >>> >>> <send> >>> >>> <endpoint key="RealService"/> >>> >>> </send> >>> >>> </then> >>> >>> <else> >>> >>> <makefault version="soap11"> >>> >>> <code xmlns:sf11="http://schemas.xmlsoap.org/soap/envelope/"; >>> value="sf11:Server"/> >>> >>> <reason value="First order must be for the symbol IBM"/> >>> >>> </makefault> >>> >>> <header name="To" action="remove"/> >>> >>> <property name="RESPONSE" value="true"/> >>> >>> <send/> >>> >>> </else> >>> >>> </filter> >>> >>> </inSequence> >>> >>> </target> >>> >>> <publishWSDL key="ProxyWSDL-embedded.wsdl"/> >>> >>> </proxy> >>> >>> >>> Here are the other improvements can be done for this method. >>> >>> Here user have to set whether it is a pass through proxy or not using a >>> property mediator. However we can use some mechanism where each synapse >>> mediator set a property if that change the incoming message. If the message >>> has not been changed that can be pass through like this. >>> >>> I have used buffered Input stream here. We may write a custom Input stream >>> reader to perform this job better. >>> >>> Currently Axiom xpath build the whole soap envelope. Therefore if we can >>> write an axiom xpath engine which only builds the required parts performance >>> can be further improved. >>> >>> I have created the patch basically demonstrate the concept. There can be >>> more efficient way of implementing as well. >>> >>> WDTY? >>> >>> Thanks, >>> >>> Amila. >>> >>> [1] https://issues.apache.org/jira/browse/SYNAPSE-909 >>> >>> >>> >>> -- >>> Amila Suriarachchi >>> WSO2 Inc. >>> blog: http://amilachinthaka.blogspot.com/ >> >> >> >> >> -- >> Hiranya Jayathilaka >> Mayhem Lab/RACE Lab; >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >> E-mail: hira...@cs.ucsb.edu; Mobile: +1 (805) 895-7443 >> Blog: http://techfeast-hiranya.blogspot.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@synapse.apache.org > For additional commands, e-mail: dev-h...@synapse.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@synapse.apache.org For additional commands, e-mail: dev-h...@synapse.apache.org
