On Fri, Aug 9, 2013 at 9:26 PM, Rajika Kumarasiri <rajika.kumaras...@gmail.com> wrote: > Yeah, actually I have started working on a new syslog consumer > implementation. As part of this I am also planning to write brand new TCP > and UDP transports(for high volume syslog message deployments). I will > report any progress as I go forward.
Why would the current UDP transport not be suitable for high volume messages? Andreas > Rajika > > > On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka > <hiranya...@gmail.com>wrote: > >> >> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <peter.wri...@six-group.com> >> wrote: >> >> > Hi Hiranja, >> > >> > Great, thanks a lot, now appears to be working correctly!! >> >> That's great. >> >> I think it would be good to have some syslog mediation support built into >> Synapse. The correct solution is probably to implement a syslog message >> builder and a formatter. There's a long standing jura issue regarding this >> as well. We'll consider this feature for a future release. >> >> Thanks, >> Hiranya >> >> > I changed my class mediator to generate the XML as you suggested, and >> then had to >> > include the namespace in the iterate definition and include it in the >> xpath expression >> > before it worked correctly. >> > >> > Best regards, >> > Peter >> > >> > ================================= >> > >> > 1. XML message returned by my class mediator >> > >> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><text xmlns=" >> http://ws.apache.org/commons/ns/payload";>This is msg number 1</text><text >> xmlns="http://ws.apache.org/commons/ns/payload";>This is msg number >> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload";>This is msg >> number 3</text></soapenv:Body></soapenv:Envelope> >> > >> > 2. synapse.xml fragment (iterator definition) >> > >> > <iterate id="syslogInterator" preservePayload="false" sequential="true" >> xmlns:m0="http://ws.apache.org/commons/ns/payload"; expression="//m0:text"> >> > <target> >> > <sequence> >> > <send> >> > <endpoint> >> > <address .../> >> > </endpoint> >> > </send> >> > </sequence> >> > </target> >> > </iterate> >> > >> > -----Original Message----- >> > From: Hiranya Jayathilaka [mailto:hiranya...@gmail.com] >> > Sent: Freitag, 9. August 2013 01:54 >> > To: u...@synapse.apache.org >> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages >> in one TCP message? >> > >> > >> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <peter.wri...@six-group.com> >> wrote: >> > >> >> Hi Hiranja, >> >> >> >> I have now implemented a class mediator that parses the (multiple) >> >> syslog plain text messages and makes an XML payload from them. >> >> I've also got an iterate mediator that splits the messages and >> >> sends them individually as JMS messages to the messaging provider. >> >> So far so good... >> >> >> >> However, the individual JMS messages are now in XML form rather >> >> than plain text (with all the XML headers that I don't want). >> >> How can I get just the plain text payload to be sent? >> >> >> >> Thanks >> >> Peter >> >> >> >> ===================== >> >> >> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual >> messages separated by NL) >> >> 20 This is msg number 1 >> >> 20 This is msg number 2 >> >> 20 This is msg number 3 >> >> >> >> 2. SOAP message logged by synapse >> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><text xmlns=" >> http://ws.apache.org/commons/ns/payload";>20 This is msg number 1 >> >> 20 This is msg number 2 >> >> 20 This is msg number 3 >> >> </text></soapenv:Body></soapenv:Envelope> >> >> >> >> 3. XML message returned by my class mediator >> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><syslogMsgs><msg>This >> is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number >> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope> >> > >> > This should create a message with multiple <text> elements. See the >> output from step 2 above. It has a single <text> element with all the >> content. You should break it down to multiple <text> elements. The plain >> text serializer of Axis2 looks for this specific <text> element when >> sending plain text messages. >> > >> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload) >> of the <text> element too. It's also required. >> > >> > Thanks, >> > Hiranya >> > >> >> >> >> 4. JMS messages sent to the messaging provider >> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><msg>This is msg >> number 1</msg></soapenv:Body></soapenv:Envelope> >> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><msg>This is msg >> number 2</msg></soapenv:Body></soapenv:Envelope> >> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><msg>This is msg >> number 3</msg></soapenv:Body></soapenv:Envelope> >> >> >> >> 5. JMS messages that I want to be sent >> >> This is msg number 1 >> >> This is msg number 2 >> >> This is msg number 3 >> >> >> >> 6. synapse.xml >> >> <definitions xmlns="http://ws.apache.org/ns/synapse";> >> >> <proxy name="proxyTcp2Jms" transports="tcp"> >> >> <target> >> >> <inSequence> >> >> <!-- Define TCP listener expects plain text (syslog) >> messages --> >> >> <property name="messageType" value="text/plain" >> scope="axis2"/> >> >> <property name="OUT_ONLY" value="true"/> >> >> <property name="TRANSPORT_HEADERS" scope="axis2" >> action="remove"/> >> >> <log level="full"/> >> >> >> >> <!-- Split any multiple syslog messages for the iterator >> --> >> >> <!-- and return as XML with child messages >> --> >> >> <class >> name="com.sixtelekurs.classMediators.syslogMsgBuilder"> >> >> <log level="full"/> >> >> </class> >> >> >> >> <!-- Iterate over any multiple messages --> >> >> <iterate id="syslogInterator" preservePayload="false" >> sequential="true" expression="//msg"> >> >> <log level="full"/> >> >> <target> >> >> <sequence> >> >> <send> >> >> <endpoint> >> >> <address >> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/> >> >> </endpoint> >> >> </send> >> >> </sequence> >> >> </target> >> >> </iterate> >> >> </inSequence> >> >> <outSequence/> >> >> <faultSequence> >> >> <log level="full" category="ERROR" separator=","/> >> >> </faultSequence> >> >> </target> >> >> <parameter name="transport.tcp.port">6060</parameter> >> >> <parameter name="transport.tcp.contentType">text/plain</parameter> >> >> </proxy> >> >> >> >> </definitions> >> >> >> >> >> >> >> >> -----Original Message----- >> >> From: Hiranya Jayathilaka [mailto:hiranya...@gmail.com] >> >> Sent: Freitag, 26. Juli 2013 20:05 >> >> To: u...@synapse.apache.org >> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog >> messages in one TCP message? >> >> >> >> You should probably apply some kind of a content transformation to >> transform this payload into a XML payload with multiple "text" elements >> (one per each syslog message). But I'm not sure whether traditional >> techniques like XSLT and XQuery support that kind of transformations. After >> all you're trying to transform based on XML text and not the XML nodes. >> That's almost always very difficult. >> >> >> >> IMO you're better off writing a custom mediator to do this >> transformation. Then you can use the iterate mediator to split the message >> into its constituents and send to the JMS queue. An even more elegant >> solution would be to write a custom message builder that knows how to parse >> the incoming batch requests and construct a proper XML payload out of it. >> >> >> >> Thanks, >> >> Hiranya >> >> >> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" < >> peter.wri...@six-group.com> wrote: >> >> >> >>> Hi, >> >>> >> >>> I got synapse working with a TCP-Listener accepting plain text messages >> >>> as input, and the messages being sent as JMS messages to IBM Websphere >> MQ. >> >>> >> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and >> I'm having a problem >> >>> because the TCP listener is receiving multiple syslog messages in one >> TCP message. >> >>> The multiple messages are being sent on as one JMS message. The >> individual >> >>> syslog messages are separated by NL and have the message length at the >> start. >> >>> >> >>> Here an example message from my log file with 8 syslog messages: >> >>> >> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: , >> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: >> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope: >> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=" >> http://schemas.xmlsoap.org/soap/envelope/";><soapenv:Body><text xmlns=" >> http://ws.apache.org/commons/ns/payload";> >> >>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat" >> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"] >> >>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New" >> BC="462" BK="842" Occured="1374760680" Reported="1374761340" >> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)" >> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured: >> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"] >> >>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat" >> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"] >> >>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat" >> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"] >> >>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 - >> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"] >> >>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462" >> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem >> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)" >> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD >> on mdzhldos02)"] >> >>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat" >> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"] >> >>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 - >> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat" >> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"] >> >>> </text></soapenv:Body></soapenv:Envelope> >> >>> >> >>> What is the recommended way of handling this? Use the Iterate-Mediator >> to >> >>> split the messages and send them individually? Do I have to use Xpath >> here, >> >>> or is it possible to use e.g. javascript? Any examples would be >> appreciated. >> >>> >> >>> Thanks for any help, >> >>> Peter >> >>> >> >>> >> >>> The content of this e-mail is intended only for the confidential use >> of the person addressed. >> >>> If you are not the intended recipient, please notify the sender and >> delete this email immediately. >> >>> Thank you. >> >> >> >> -- >> >> Hiranya Jayathilaka >> >> Mayhem Lab/RACE Lab; >> >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >> >> E-mail: hira...@cs.ucsb.edu; Mobile: +1 (805) 895-7443 >> >> Blog: http://techfeast-hiranya.blogspot.com >> >> >> >> The content of this e-mail is intended only for the confidential use of >> the person addressed. >> >> If you are not the intended recipient, please notify the sender and >> delete this email immediately. >> >> Thank you. >> > >> > -- >> > Hiranya Jayathilaka >> > Mayhem Lab/RACE Lab; >> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu >> > E-mail: hira...@cs.ucsb.edu; Mobile: +1 (805) 895-7443 >> > Blog: http://techfeast-hiranya.blogspot.com >> > >> > The content of this e-mail is intended only for the confidential use of >> the person addressed. >> > If you are not the intended recipient, please notify the sender and >> delete this email immediately. >> > Thank you. >> >> -- >> Hiranya Jayathilaka >> Mayhem Lab/RACE Lab; >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >> E-mail: hira...@cs.ucsb.edu; Mobile: +1 (805) 895-7443 >> Blog: http://techfeast-hiranya.blogspot.com >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@synapse.apache.org For additional commands, e-mail: dev-h...@synapse.apache.org
