Don't change cache and buffer sizes unless you know what is happening, the
defaults are going to be fine.
How much heap did you give ES?
I'm not sure you can do much about the date filter though, maybe someone
else has pointers.
On 19 February 2015 at 21:12, Deva Raj <devarajcse...@gmail.com> wrote:
> Hi Mark Walkom,
>
> I have given below logstash conf file
>
>
> Logstash conf
>
> input {
> file {
>
> }
>
> }
>
> filter {
> mutate
> {
> gsub => ["message", "\n", " "]
> }
> mutate
> {
> gsub => ["message", "\t", " "]
> }
> multiline
> {
> pattern => "^ "
> what => "previous"
> }
>
> grok { match => [ "message",
> "%{TIME:log_time}\|%{WORD:Message_type}\|%{GREEDYDATA:Component}\|%{NUMBER:line_number}\|
> %{GREEDYDATA:log_message}"]
> match => [ "path" ,
> "%{GREEDYDATA}/%{GREEDYDATA:loccode}/%{GREEDYDATA:_machine}\:%{DATE:logdate}.log"]
>
> break_on_match => false
> }
>
>
> #To check location is S or L
> if [loccode] == "S" or [loccode] == "L" {
> ruby {
> code => " temp = event['_machine'].split('_')
> if !temp.nil? || !temp.empty?
> event['_machine'] = temp[0]
> end"
> }
> }
> mutate {
>
> add_field => ["event_timestamp", "%{@timestamp}" ]
> replace => [ "log_time", "%{logdate} %{log_time}" ]
> # Remove the 'logdate' field since we don't need it anymore.
> lowercase=>["loccode"]
> remove => "logdate"
>
> }
> # to get all site details (site name, city and co-ordinates)
> sitelocator{sitename => "loccode"
> datafile=>"vendor/sitelocator/SiteDetails.csv"}
> date { locale=>"en"
> match => [ "log_time", "yyyy-MM-dd HH:mm:ss", "MM-dd-yyyy
> HH:mm:ss.SSS","ISO8601" ] }
>
> }
>
> output {
> elasticsearch{
> }
>
> }
>
>
>
> I have checked step by step to find bottleneck filter. Below filter which
> took much time. Can you guide me How can I tune it to get faster.
>
> date { locale=>"en" match => [ "log_time", "yyyy-MM-dd HH:mm:ss",
> "MM-dd-yyyy HH:mm:ss.SSS","ISO8601" ] } }
> <http://serverfault.com/questions/669534/elasticsearch-performance-tuning#comment818613_669558>
>
>
> Thanks
> Devaraj
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7eedf369-b10d-442e-b30d-5e7969bf1c59%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/7eedf369-b10d-442e-b30d-5e7969bf1c59%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-bB8%3DY0fd4HKcJ9Tw6OENwOTkMYo2muZs-Pd7-dt%2BA9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
