As requested here is a dump of the hot threads output.
Thanks!
J
On Thursday, February 12, 2015 at 6:45:23 PM UTC-5, Nikolas Everett wrote:
>
> You might want to try hitting hot threads while putting your load on it
> and seeing what you see. Or posting it.
>
> Nik
>
> On Thu, Feb 12, 2015 at 4:44 PM, Jay Danielian <jay.da...@circleback.com
> <javascript:>> wrote:
>
>> Mark,
>>
>> Thanks for the initial reply. Yes, your assumption about these things
>> being very specific and thus not likely to have any re-use with regards to
>> caching is correct. I have attached some screenshots from the BigDesk
>> plugin which showed a decent snapshot of what the server looked like while
>> my tests were running. You can see the spikes in CPU, that essentially
>> covered the duration when the JMeter tests were running.
>>
>> At a high level, the only thing that seems to be really stressed on the
>> server is CPU. But that makes me think that there is something in my setup
>> , query syntax, or perhaps the cache eviction rate, etc that is causing it
>> to spike so high. I also have concerns about non RAID 0 the EBS volumes, as
>> I know that having one large volume does not maximize throughput - however,
>> just looking at the stats it doesn't seem like IO is really a bottleneck.
>>
>> Here is a sample query structure =>
>> https://gist.github.com/jaydanielian/c2be885987f344031cfc
>>
>> Also this is one query - in reality we use _msearch to pipeline several
>> of these queries in one batch. The queries also include custom routing /
>> route key to make sure we only hit one shard.
>>
>> Thanks!
>>
>> J
>>
>>
>> On Thursday, February 12, 2015 at 4:22:29 PM UTC-5, Mark Walkom wrote:
>>>
>>> It'd help if you could gist/pastebin/etc a query example.
>>>
>>> Also your current ES and java need updating, there are known issues with
>>> java <1.7u55, and you will always see performance boosts running the latest
>>> version of ES.
>>>
>>> That aside, what is your current resource utilisation like? Are you
>>> seeing lots of cache evictions, high heap use, high CPU, IO delays?
>>>
>>> On 13 February 2015 at 07:32, Jay Danielian <jay.da...@circleback.com>
>>> wrote:
>>>
>>>> I know this is difficult to answer, the real answer is always "It
>>>> Depends" :) But I am going to go ahead and hope I get some feedback here.
>>>>
>>>> We are mainly using ES to issue terms searches against fields that are
>>>> non-analyzed. We are using ES like a key value store, where once the match
>>>> is found we parse the _source JSON and return our model. We are doing
>>>> contact lookups, searching against (last_name AND (phone_number OR
>>>> email)).
>>>> We are issuing constant_score queries with term filters for the terms
>>>> mentioned above. No aggregations, no sorting, no scripts, etc. Using
>>>> JMeter, we were maxing out at around 500 search requests / sec. Average
>>>> request time was taking around 7 seconds to complete. When the test would
>>>> fire up, the ThreadPool Search Queue would spike to 1000 on each node and
>>>> CPU would be maxed out, then once it finished everything would return to
>>>> normal. So it appears healthy, and we wouldn't get any errors - just
>>>> nowhere close to the performance we are looking for.
>>>>
>>>> Setup details
>>>> - Index size 100GB with two different document mappings in the index.
>>>> Roughly 500M documents
>>>> - three nodes c3.4xl instances on EC2 using pIOPS SSD EBS disks
>>>> (although NOT RAID 0 - just one big volume)
>>>> - each server node on EC2 has 30GB RAM, 16GB on heap, rest for OS
>>>> - we have set mlockall on our instances
>>>> - 3 nodes are split into 6 shards for the main index
>>>> - Index is read only after it is loaded - we don't update the index
>>>> ever, it is only for querying
>>>> - ES version 1.3.3 Java 1.7.0_51
>>>> - each server has 16 cores / node and 48 search threads with queue
>>>> length of 1000
>>>>
>>>> Assuming no stemming, free text queries - just term matching, how can
>>>> we increase the throughput and decrease the response time for the ES
>>>> queries? is 500 requests / sec at the top end?
>>>> Do we just need many more servers if we really want 3000 requests / sec
>>>> ? I have read that scaling out is better for ES vs scaling up. But it
>>>> feels
>>>> that the current server farm should deliver better performance.
>>>>
>>>> Any help or tuning advice would be really appreciated. We have looked
>>>> at many slideshares, blog posts from found.no, elasticseearch.org, etc
>>>> - and can't really pinpoint a way to improve our setup.
>>>>
>>>> Thanks!
>>>>
>>>> JD
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elasticsearc...@googlegroups.com.
>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>> msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%
>>>> 40googlegroups.com
>>>> <https://groups.google.com/d/msgid/elasticsearch/47b93b84-d929-4cad-becd-31581cd4c574%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/3d6b37eb-7ec1-47d7-bcb4-df96203b1445%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elasticsearch/3d6b37eb-7ec1-47d7-bcb4-df96203b1445%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/476bb853-85d6-4cd5-81f7-a31d19a09c7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
:::
[siq-prod-es01][Azyzwh_wQzqRIwsUNhfWQw][siq-es-307f91d1][inet[/10.10.80.40:9300]]{aws_availability_zone=us-east-1e,
max_local_storage_nodes=1}
20.5% (102.3ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es01][search][T#11]'
2/10 snapshots sharing following 31 elements
org.apache.lucene.queries.TermsFilter.getDocIdSet(TermsFilter.java:206)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
8/10 snapshots sharing following 32 elements
org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:189)
org.apache.lucene.queries.TermsFilter.getDocIdSet(TermsFilter.java:200)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
15.1% (75.5ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es01][search][T#46]'
8/10 snapshots sharing following 32 elements
org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:189)
org.apache.lucene.queries.TermsFilter.getDocIdSet(TermsFilter.java:200)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
2/10 snapshots sharing following 30 elements
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
13.5% (67.5ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es01][search][T#14]'
10/10 snapshots sharing following 32 elements
org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:189)
org.apache.lucene.queries.TermsFilter.getDocIdSet(TermsFilter.java:200)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
:::
[siq-prod-es03][ShRjOG2wS_esL2lyymZbng][siq-es-0b73cbea][inet[/10.10.80.88:9300]]{aws_availability_zone=us-east-1e,
max_local_storage_nodes=1}
61.3% (306.3ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es03][search][T#41]'
10/10 snapshots sharing following 29 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
48.2% (241.1ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es03][search][T#19]'
10/10 snapshots sharing following 29 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
45.0% (224.9ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es03][search][T#13]'
10/10 snapshots sharing following 29 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
:::
[siq-prod-es02][kmB9ibKiQXe73fp4THgbUg][siq-es-bc248756][inet[/10.10.80.227:9300]]{aws_availability_zone=us-east-1d,
max_local_storage_nodes=1}
69.1% (345.2ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es02][search][T#4]'
4/10 snapshots sharing following 41 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.OrDocIdSet$IteratorBasedIterator.<init>(OrDocIdSet.java:122)
org.elasticsearch.common.lucene.docset.OrDocIdSet.iterator(OrDocIdSet.java:73)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
6/10 snapshots sharing following 40 elements
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:50)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.OrDocIdSet$IteratorBasedIterator.<init>(OrDocIdSet.java:122)
org.elasticsearch.common.lucene.docset.OrDocIdSet.iterator(OrDocIdSet.java:73)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
57.0% (285ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es02][search][T#30]'
3/10 snapshots sharing following 39 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
7/10 snapshots sharing following 22 elements
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
43.3% (216.7ms out of 500ms) cpu usage by thread
'elasticsearch[siq-prod-es02][search][T#13]'
8/10 snapshots sharing following 39 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
2/10 snapshots sharing following 41 elements
sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
sun.nio.ch.IOUtil.read(IOUtil.java:197)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:699)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:207)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs(Lucene41PostingsReader.java:380)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:422)
org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:84)
org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:130)
org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:163)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.AndDocIdSet$IteratorBasedIterator.newDocIdSetIterator(AndDocIdSet.java:146)
org.elasticsearch.common.lucene.docset.AndDocIdSet.iterator(AndDocIdSet.java:92)
org.apache.lucene.search.FilteredDocIdSet.iterator(FilteredDocIdSet.java:96)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter$WrappedDocIdSet.iterator(ApplyAcceptedDocsFilter.java:199)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:161)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:335)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:751)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:740)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
