You will never get around the spike on some level. Elasticsearch uses resources at index time to ensure the searching is fast.
You may be able to reduce it a little by playing with your bulk size, ie reducing it, or adding more CPUs On 11 February 2015 at 08:23, Sagar Shah <[email protected]> wrote: > Thanks for your input Mark. > > Can that be optimized? Does that run real time? I configured > index.refresh_interval to run every 30 seconds. Can I optimize indexing > process anyway. > I have observed that whenever I process 3-4 events in my application > causing about 5-6K records insertion in elasticseach, there's cpu spike. > > I monitored heap as well, and its way below (200mb) size configured (2gb) > in ES. > > I don't need to use any fancy feature of cluster or availability at this > point. > My whole objective (from previous post) is to ensure that I store logs > information and retrieve it using API and show in my web administration > application without causing any performance or cpu spike. > > Please advise. > Thanks! > > Regards, > Sagar Shah > On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote: >> >> That would be the indexing process that happens when you send the logs, >> as ES needs to process and turn these into an inverted index for searching. >> >> On 11 February 2015 at 07:23, Sagar Shah <[email protected]> wrote: >> >>> Hello everyone, >>> I am new to Elastic Search world and I am evaluating it for storing and >>> retrieval of my application logs, which is currently being stored in >>> postgres database. >>> For performance and disk space reasons, I plan to switch logs storage >>> from posgres to elastic search. I plan to store log information for various >>> applications as different indices in ES. >>> >>> But when I implemented this tool, I noticed lots of cpu spike on my >>> system when it received the request for inserting logs data into ES. >>> I am using Linux and allocated ES a 2GB RAM. I am using REST service >>> calls to push logs data into ES. There are about 5000 records being >>> inserted into one index for every request into the application (all in JSON >>> format). Refresh interval is kept to 30 seconds in elasticsearch.yml file >>> >>> Number of shards configured to 1 and replicas to 0. There are about 20 >>> index created for different applications, which might eventually grow upto >>> 50 (as it is per application) Also, advise, if it should be separate index >>> per application, or single index storing logs of all of the applications. I >>> expect about 3k to 5k log messages per any request in any application >>> Single node is being used for storing logs of multiple applications >>> running on same node under separate JVM >>> JDK: 1.7 >>> >>> >>> ES_HEAP_SIZE=2g >>> Also configured, mlockall to true in elasticsearch.yml file. >>> >>> I am clueless in identifying root cause of cpu spike. >>> >>> Can anyone please suggest a way out in further troubleshooting and >>> determining the root cause. What should be the next steps to troubleshoot >>> the same. >>> >>> ES version: 1.4.2 >>> >>> Please let me know, if you need any further information. >>> Appreciate your inputs and help! >>> >>> Regards, >>> Sagar Shah >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
