Respected Sir, Removed the logger, but still not able to process large files. Requirement is to process files of size upto GB's . Please Guide.
With Regards, Aditya On Wed, Mar 14, 2018 at 8:08 PM, Ruwan Abeykoon <ruw...@wso2.com> wrote: > Hi Adithya, > Can you try this without the log mediator. > Log mediator is usually tries to decode the payload. We call that a > content aware mediator. The content aware mediators has the issue decoding > large payloads. > > Cheers, > Ruwan > > On Wed, Mar 14, 2018 at 6:43 PM, aditya shivankar < > shivankar.adit...@gmail.com> wrote: > >> Respected Sir, >> >> Forgot to mention ,Sometimes I am getting OutOfMemoryError as well >> >> java.lang.OutOfMemoryError: Java heap space >> Dumping heap to C:/wso2ei-6.1.1/wso2ei-6.1.1/repository/logs/heap-dump.hprof >> ... >> Unable to create C:/wso2ei-6.1.1/wso2ei-6.1.1/r >> epository/logs/heap-dump.hprof: File exists >> [2018-03-14 17:16:01,228] [EI-Core] ERROR - NativeWorkerPool Uncaught >> exception >> java.lang.OutOfMemoryError: Java heap space >> at java.util.Arrays.copyOf(Unknown Source) >> at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source) >> at java.lang.AbstractStringBuilder.append(Unknown Source) >> at java.lang.StringBuffer.append(Unknown Source) >> at org.apache.log4j.helpers.PatternParser$LiteralPatternConvert >> er.format(PatternParser.java:419) >> >> With Regards, >> Aditya >> >> On Wed, Mar 14, 2018 at 4:46 PM, aditya shivankar < >> shivankar.adit...@gmail.com> wrote: >> >>> added the missing attachment from previous mail >>> >>> On Wed, Mar 14, 2018 at 4:44 PM, aditya shivankar < >>> shivankar.adit...@gmail.com> wrote: >>> >>>> Respected Sir/Madam, >>>> >>>> I am able to achieve below file processing using vfs, and datamapper. >>>> But when trying to process files more than size of 8 MB or more the flow is >>>> failing with out of memory. Please guide >>>> >>>> step 1. I am using vfs to read one xml file from one location. >>>> step 2. Then using datamapper to convert the xml file to csv. >>>> step 3. Then writing the converted output to another location. >>>> >>>> >>>> <?xml version="1.0" encoding="UTF-8"?> >>>> <proxy name="FlatFileProxy" startOnLoad="true" transports="vfs" xmlns=" >>>> http://ws.apache.org/ns/synapse";> >>>> <target> >>>> <inSequence> >>>> <log level="custom"> >>>> <property name="property_name" value=""Inside >>>> FlatFile ***************************************** ""/> >>>> </log> >>>> <property name="FORCE_SC_ACCEPTED" scope="axis2" >>>> type="STRING" value="true"/> >>>> <property name="REST_URL_POSTFIX" scope="axis2" >>>> type="STRING" value=""/> >>>> <datamapper config="gov:datamapper/xmlToCsv1.dmc" >>>> inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json" >>>> inputType="XML" outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json" >>>> outputType="CSV"/> >>>> <property expression="fn:concat(fn:subst >>>> ring-after(get-property('MessageID'), 'urn:uuid:'), '.csv')" >>>> name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/> >>>> <property name="OUT_ONLY" scope="default" type="STRING" >>>> value="true"/> >>>> <property name="messageType" scope="axis2" type="STRING" >>>> value="application/csv"/> >>>> <property action="remove" name="LAST_MODIFIED" >>>> scope="transport"/> >>>> <send> >>>> <endpoint> >>>> <address uri="vfs:file:///C:/Flatfile/output"/> >>>> </endpoint> >>>> </send> >>>> </inSequence> >>>> <outSequence/> >>>> <faultSequence/> >>>> </target> >>>> <parameter name="transport.PollInterval">50ms</parameter> >>>> <parameter name="transport.vfs.FileURI">f >>>> ile:///C:/Flatfile/input</parameter> >>>> <parameter name="transport.vfs.ContentTyp >>>> e">application/xml</parameter> >>>> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter> >>>> <parameter name="transport.vfs.MoveAfterF >>>> ailure">file:///C:/Flatfile/failure</parameter> >>>> <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter> >>>> <parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter> >>>> <parameter name="transport.vfs.Locking">disable</parameter> >>>> <parameter name="transport.vfs.MoveAfterP >>>> rocess">file:///C:/Flatfile/orgFilesProcessedSuccessfully</parameter> >>>> </proxy> >>>> >>>> >>>> sometimes it is not able to delete .lock files from input folder. >>>> sometimes timeout error. >>>> >>>> etc.. >>>> >>>> Sample input attached. >>>> >>>> With regards, >>>> Aditya >>>> >>> >>> >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
