Camel Tracer and Logstash
Has anyone managed to integrate Camel logs with Logstash/Elasticsearch?
I've got tracer and MDC logging enabled in my CamelContext and am sending
logs to Logstash via log4j TCP appender. This is working great (especially
MDC fields), but I'm trying to further parse and filter the message to
individual fields in Elasticsearch.
For example, my message is:
ID-dx123abc001-abc-com-42264-1475782902034-0-193168 >>> (system-sql)
setHeader[Content-Type, application/json] -->
marshal[org.apache.camel.component.jackson.JacksonDataFormat@7e38f55] <<<
Pattern:InOut, Headers:{Accept-Encoding=gzip, deflate, Authorization=Basic
XRpeDpreG1pZGFz,
breadcrumbId=ID-dx123abc001-abc-com-42264-1475782902034-0-193169,
CamelHttpMethod=GET, CamelHttpPath=, CamelHttpQuery=name=new-strategies,
CamelHttpServletRequest=Request(GET
/esb/system/sql?name=new-strategies)@6aaa8b00,
CamelHttpServletResponse=HTTP/1.1 200 , CamelHttpUri=/esb/system/sql,
CamelHttpUrl=http://system.abc.com:51950/esb/system/sql,
CamelJdbcColumnNames=[STRATEGY_NAME], CamelJdbcRowCount=2,
CamelServletContextPath=/esb/system/sql, Connection=keep-alive,
Content-Type=application/json, Host=system.abc.com:51950,
name=new-strategies, User-Agent=Apache-HttpClient/4.2.1 (java 1.5)},
BodyType:java.util.ArrayList, Body:[{STRATEGY_NAME=S1 }, {STRATEGY_NAME=S2
}]
And I would like logstash to parse and filter this so that it is stored in
Elasticsearch as individual fields:
...
from=setHeader[Content-Type, application/json]
to=marshal[org.apache.camel.component.jackson.JacksonDataFormat@7e38f55]
breadcrumbId=ID-dx123abc001-abc-com-42264-1475782902034-0-193169
CamelJdbcColumnNames=[STRATEGY_NAME]
CamelJdbcRowCount=2
BodyType:java.util.ArrayList
Body:[{STRATEGY_NAME=S1 }, {STRATEGY_NAME=S2 }]
etc.
Has anyone been able to do this? I'm struggling with getting this to work
with grok or ruby in the logstash config. Any tips are appreciated!
Thanks
--
View this message in context:
http://camel.465427.n5.nabble.com/Camel-Tracer-and-Logstash-tp5788502.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Skip duplicate for idempotent file2 component
I am using camel over a clustered environment and want to use a readlock on my file consumer endpoint. The only cluster safe readlock is the idempotent readlock however this requires an idempotentRepository to be set on the file uri. I use an idempotent consumer within the route which moves any duplicate files to an error folder and logs the error to a specific file. Is there a way to use this idempotent consumer with the idempotent readlock? Or a way to set the idempotentRepository in the file component to not skip the duplicates so they are picked up by the idempotent consumer in the route instead? -- View this message in context: http://camel.465427.n5.nabble.com/Skip-duplicate-for-idempotent-file2-component-tp5788485.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: syslog dateformat + log4j syslog appender
The syslog date format in log4j 2 is "MMM dd HH:mm:ss" (SimpleDateFormat
style).
On 7 October 2016 at 04:09, fabryprog wrote:
> Hello everyone,
>
> I'm trying to figure out how to set log4j conversionPattern to be
> compatible
> with the SyslogDateFormat.
>
> I try without results:
>
> log4j.appender.syslog.layout.conversionPattern=%d{ABSOLUTE} %-5p [%c{1}]
> %m
>
> log4j.appender.syslog.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2}
> %x - %m%n
>
> Can you help me?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/syslog-dateformat-log4j-syslog-appender-tp5788476.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Matt Sicker
camel SCP tempPrefix option won't work
Hello,
This is my camel scp test code.
I'd like to send a file to remote server with prefix "temp_".
After finishing scp file copy, the "temp_" prefix is supposed to be gone.
BUT the file name with tempPrefix never work. While writing, the file is
same with original file.
Did I miss something?
Camel version is 2.17.3 and I'm testing on jdk 1.8.
Thanks.
from("file:" + raw_dir + "?delete=true")
.to("scp:xxx.xxx.xxx.xxx/data/scptest?tempPrefix=temp_&username=user&password=passwd&knownHostsFile=/Users/skeom21/.ssh/known_hosts&strictHostKeyChecking=no")
.log("${headers.CamelFileName}")
.end();
--
View this message in context:
http://camel.465427.n5.nabble.com/camel-SCP-tempPrefix-option-won-t-work-tp5788465.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Calling {body.getXXXXXX}from Simple EL taking > 200 ms
Can i have a test code or test mock project? -- View this message in context: http://camel.465427.n5.nabble.com/Calling-body-getXX-from-Simple-EL-taking-200-ms-tp5788478p5788481.html Sent from the Camel - Users mailing list archive at Nabble.com.
Calling {body.getXXXXXX}from Simple EL taking > 200 ms
Hi
I notice that when making method calls on the body using the Simple EL they
are quite slow depending on the size of the object on the exchange, e.g.
${body.getXX}
Is there anyway to speed this up?
Thanks
Joe
--
View this message in context:
http://camel.465427.n5.nabble.com/Calling-body-getXX-from-Simple-EL-taking-200-ms-tp5788478.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File Expression Language & date format
Excuse me. Maybe I did not quite understand spoken. Or someone does not quite
understand.
On page http://camel.apache.org/file-language.html
file:modified - Refers to the file last modified returned as a Date type.
Writing
idempotentKey=${file:name}-${file:size}-${file:modified}
"file:modified" type is Long. Class GenericFile, field long lastModified;
If the "file:modified" is an introspection on the lastModified, then the
returned data type Long.
Or, do you want to do "file:modified" ==> new Date(file.getLastModified())
Best regards,
Aleksey
--
View this message in context:
http://camel.465427.n5.nabble.com/File-Expression-Language-date-format-tp5788388p5788477.html
Sent from the Camel - Users mailing list archive at Nabble.com.
syslog dateformat + log4j syslog appender
Hello everyone,
I'm trying to figure out how to set log4j conversionPattern to be compatible
with the SyslogDateFormat.
I try without results:
log4j.appender.syslog.layout.conversionPattern=%d{ABSOLUTE} %-5p [%c{1}] %m
log4j.appender.syslog.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2}
%x - %m%n
Can you help me?
--
View this message in context:
http://camel.465427.n5.nabble.com/syslog-dateformat-log4j-syslog-appender-tp5788476.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel-Mail - Consume from IMAP with variable searchTerm.fromSentDate
Unfortunately it is not sufficient to provide own SearchTerm class and overriding "match" because com.sun.mail.imap.protocol.SearchSequence checks specific implementations of SearchTerm provided by com.sun.mail and all implementations are final. I worked around the issue by resetting the searchTerm in Camel MailEndpoint everytime I want to update the criteria. This works as MailConsumer retrieves the searchTerm from the endpoint on each poll. 2016-10-06 13:21 GMT+02:00 Claus Ibsen : > I think you just need to keep state yourself in the bean, and then the > mail component will call the match method to see if a mail messages > matches or not. > > Then your state can then update with new sent date information so the > matches method can use a newer date. > > On Thu, Oct 6, 2016 at 2:43 AM, Michael Täschner > wrote: > > Hi Again, > > > > is there a possibility that the searchTerm bean reference picked up by > > camel-mail can update it's internal state to allow for a changing > sentDate > > for the query? > > > > Thanks and Best Regards, > > Michael > > > > 2016-09-23 16:21 GMT+02:00 Michael Täschner : > > > >> Hi Camel-Riders, > >> > >> is there a possibility to consume mails with modifiable > >> searchTerm.fromSentDate value? I am looking at a solution where the > >> consumer must not depend on "unseen" flag, manages an internal dateTime > >> variable to pick-up mail search from where it last stopped? There may > be > >> other clients on the mail account and I cannot depend on the "unseen" > flag. > >> Is this a case for poll-enrich or how can I manipulate the from query to > >> use the variable "searchTerm.fromSentDate"? > >> > >> Thanks and Best Regards, > >> Michael > >> > > > > -- > Claus Ibsen > - > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
