Re: My Elasticsearch is running at very high CPU (constantly 99%) - Need help understanding hot_threads
Do you run many segments in your shards? If so, you should consider to run optimize. Jörg On Mon, Aug 4, 2014 at 6:46 PM, Melanie Zamora wrote: > Hello, > > My ES node is running very high and this is the thread from running > hot_threads. Can someone help me understand why CPU is so high? > > This is a 1 node cluster running a linux machine with 16 cores and 72GB of > RAM. > > ::: [c9q9o027][VTde-tWsTcm9oe7vBZuadA][inet[/10.1.6.21:9300]] > >48.3% (241.3ms out of 500ms) cpu usage by thread > 'elasticsearch[c9q9o027][search][T#3]' > 10/10 snapshots sharing following 17 elements > > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:245) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:627) > > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269) >org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119) > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233) > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202) > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80) > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203) > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186) >java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >java.lang.Thread.run(Unknown Source) > >48.3% (241.3ms out of 500ms) cpu usage by thread > 'elasticsearch[c9q9o027][search][T#8]' > 2/10 snapshots sharing following 23 elements > > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:190) > > org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:58) > > org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:54) > > org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143) > > org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134) > > org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:69) > > org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618) > > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281) >org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269) >org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119) > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233) > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202) > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80) > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203) > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186) >java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >java.lang.Thread.run(Unknown Source) > 8/10 snapshots sharing following 23 elements > > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:190) > > org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:61) > > org.apache.lucene
My Elasticsearch is running at very high CPU (constantly 99%) - Need help understanding hot_threads
Hello, My ES node is running very high and this is the thread from running hot_threads. Can someone help me understand why CPU is so high? This is a 1 node cluster running a linux machine with 16 cores and 72GB of RAM. ::: [c9q9o027][VTde-tWsTcm9oe7vBZuadA][inet[/10.1.6.21:9300]] 48.3% (241.3ms out of 500ms) cpu usage by thread 'elasticsearch[c9q9o027][search][T#3]' 10/10 snapshots sharing following 17 elements org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:245) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:627) org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119) org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233) org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202) org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80) org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203) org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186) java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.lang.Thread.run(Unknown Source) 48.3% (241.3ms out of 500ms) cpu usage by thread 'elasticsearch[c9q9o027][search][T#8]' 2/10 snapshots sharing following 23 elements org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:190) org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:58) org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:54) org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143) org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134) org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:69) org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618) org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269) org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119) org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233) org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202) org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80) org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203) org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186) java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.lang.Thread.run(Unknown Source) 8/10 snapshots sharing following 23 elements org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:190) org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:61) org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:53) org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143) org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134) org.elasticsearch.common.lucene.search.FilteredCollect
