Hi Mark,
Thanks for the reply. We are using ES 0.90 on Java 7
The stats are as under :
"ok" : true,
"_shards" : {
"total" : 300,
"successful" : 300,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 28403397,
"deleted" : 1385142
},
"store" : {
"size" : "465.4gb",
"size_in_bytes" : 499761991425,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 2343265,
"index_time" : "13.9d",
"index_time_in_millis" : 1204844492,
"index_current" : 0,
"delete_total" : 5,
"delete_time" : "659ms",
"delete_time_in_millis" : 659,
"delete_current" : 0
},
"get" : {
"total" : 9,
"get_time" : "3.2s",
"time_in_millis" : 3277,
"exists_total" : 6,
"exists_time" : "3.2s",
"exists_time_in_millis" : 3277,
"missing_total" : 3,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 32,
"query_total" : 262779551,
"query_time" : "430.4d",
"query_time_in_millis" : 37193583386,
"query_current" : 3,
"fetch_total" : 115208512,
"fetch_time" : "541.6d",
"fetch_time_in_millis" : 46798459635,
"fetch_current" : 9
}
},
"total" : {
"docs" : {
"count" : 56806794,
"deleted" : 2770284
},
"store" : {
"size" : "930.8gb",
"size_in_bytes" : 999509417525,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 4682044,
"index_time" : "34.6d",
"index_time_in_millis" : 2993312068,
"index_current" : 2232,
"delete_total" : 10,
"delete_time" : "2.4s",
"delete_time_in_millis" : 2404,
"delete_current" : 0
},
"get" : {
"total" : 24,
"get_time" : "9.8s",
"time_in_millis" : 9855,
"exists_total" : 15,
"exists_time" : "9.8s",
"exists_time_in_millis" : 9855,
"missing_total" : 9,
"missing_time" : "0s",
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 64,
"query_total" : 525308020,
"query_time" : "816.1d",
"query_time_in_millis" : 70519558277,
"query_current" : 6,
"fetch_total" : 230215915,
"fetch_time" : "939.6d",
"fetch_time_in_millis" : 81182884575,
"fetch_current" : 10
}
}
}
I am attaching along the hot_threads output, though I couldn't make much
sense of it. Pardon me if I sound N00Bish, because I am pretty new to ES.
On Wednesday, May 14, 2014 11:27:07 AM UTC+5:30, Mark Walkom wrote:
>
> ES will use descriptors to open files it needs for general operations and
> it can have a lot of those open at one time. But the level you've set and
> the level you're using are not related to your slow queries.
>
> How much data do you have in your cluster? What version of ES, what
> version of java? You should also check your slow log
> (/var/log/elasticsearch/*_index_search_slowlog.log) and hot threads (GET
> /_cluster/nodes/hot_threads) to see what is happening.
> Also try installing a plugin like Marvel or ElasticHQ to give you a better
> idea of what your resource usage is like on a cluster level. Bigdesk is
> good but it limited to a node only view.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com <javascript:>
> web: www.campaignmonitor.com
>
>
> On 14 May 2014 15:47, Tamanjit Bindra <bindra....@gmail.com
> <javascript:>>wrote:
>
>> Hi,
>> I have read
>> here<http://www.elasticsearch.org/tutorials/too-many-open-files/>,
>> that we should set the soft and hard values of nooffiles for ES to utilise.
>>
>> I have done that, but somehow my BigDesk console shows that even though
>> the upper limit is 65535, ES is only using about 2K+ file descriptors at
>> any given moment.
>>
>> I have set the ulimit -Hn and ulimit -Sn to 32000.
>>
>> Why is this so?
>>
>> I ask this because my queries are performing slow.
>>
>> We are using 2 nodes. Both servers have 16G RAM with 10G dedicated to ES.
>>
>> Please Help!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c27bd6d3-28a4-470b-96db-08117f554b7d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d854a1e6-20a0-4424-86b0-2322f18b6047%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
::: [xxx-212][GnPBS12eRGOTJkIPurDf3g][inet[/searchserver:9200]]
1.8% (8.8ms out of 500ms) cpu usage by thread
'elasticsearch[xxx-212][search][T#16]'
10/10 snapshots sharing following 8 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
0.7% (3.7ms out of 500ms) cpu usage by thread
'elasticsearch[xxx-212][search][T#11]'
8/10 snapshots sharing following 8 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
2/10 snapshots sharing following 25 elements
sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:51)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:215)
sun.nio.ch.IOUtil.read(IOUtil.java:191)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:674)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:272)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:51)
org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:181)
org.apache.lucene.store.DataInput.readLong(DataInput.java:131)
org.apache.lucene.store.BufferedIndexInput.readLong(BufferedIndexInput.java:194)
org.apache.lucene.codecs.lucene3x.Lucene3xStoredFieldsReader.visitDocument(Lucene3xStoredFieldsReader.java:226)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:139)
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:116)
org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:215)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:144)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:434)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
0.5% (2.3ms out of 500ms) cpu usage by thread
'elasticsearch[xxx-212][search][T#15]'
8/10 snapshots sharing following 8 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
2/10 snapshots sharing following 25 elements
sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:51)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:215)
sun.nio.ch.IOUtil.read(IOUtil.java:191)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:674)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:272)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:51)
org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:181)
org.apache.lucene.store.DataInput.readLong(DataInput.java:131)
org.apache.lucene.store.BufferedIndexInput.readLong(BufferedIndexInput.java:194)
org.apache.lucene.codecs.lucene3x.Lucene3xStoredFieldsReader.visitDocument(Lucene3xStoredFieldsReader.java:226)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:139)
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:116)
org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:215)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:144)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:434)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
::: [xxx-213][m2G-4PBeRpelQndL-_qr_g][inet[/172.20.20.213:9300]]{master=true}
4.0% (20ms out of 500ms) cpu usage by thread
'elasticsearch[xxx-213][search][T#12]'
9/10 snapshots sharing following 17 elements
org.apache.lucene.search.Scorer.score(Scorer.java:63)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:605)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:161)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:482)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:438)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:239)
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:141)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:679)
unique snapshot
sun.nio.ch.FileDispatcher.pread0(Native Method)
sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:49)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:248)
sun.nio.ch.IOUtil.read(IOUtil.java:224)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:663)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:272)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:51)
org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:181)
org.apache.lucene.store.DataInput.readLong(DataInput.java:131)
org.apache.lucene.store.BufferedIndexInput.readLong(BufferedIndexInput.java:194)
org.apache.lucene.codecs.lucene3x.Lucene3xStoredFieldsReader.visitDocument(Lucene3xStoredFieldsReader.java:226)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:139)
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:116)
org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:215)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:144)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:434)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:634)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:623)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:679)
2.0% (10ms out of 500ms) cpu usage by thread
'elasticsearch[xxx-213][transport_client_worker][T#3]{New I/O worker #3}'
9/10 snapshots sharing following 15 elements
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:679)
unique snapshot
sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:274)
sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:284)
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:679)
2.0% (10ms out of 500ms) cpu usage by thread
'elasticsearch[xxx-213][search][T#1]'
10/10 snapshots sharing following 34 elements
sun.nio.ch.FileDispatcher.pread0(Native Method)
sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:49)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:248)
sun.nio.ch.IOUtil.read(IOUtil.java:224)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:663)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:272)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:51)
org.apache.lucene.store.DataInput.readVInt(DataInput.java:108)
org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:219)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2357)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1693)
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:167)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:401)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:401)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:82)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:664)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:124)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:239)
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:141)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:679)
