So, your "search-only" machines are running out of memory, while your "index-only" machines are doing fine. Did I understand you correctly? Could you send me nodes stats (curl "localhost:9200/_nodes/stats?pretty") from the machine that runs out of memory, please run stats a few times with 1 hour interval. I would like to see how memory consumption is increasing over time. Please, also run nodes info ones (curl "localhost:9200/_nodes") and post here (or send me by email) the results. Thanks!
On Wednesday, July 2, 2014 10:15:46 AM UTC-4, JoeZ99 wrote: > > Hey, Igor, thanks for answering! and sorry for the delay. Didn't catch the > update. > > I explain: > > - we have one cluster of one machine which is only meant for serving > search requests. the goal is not to index anything to it. It contains 1.7k > indices, give it or take it. > - every day, those 1.7k indices are reindexed, and snapshoted in pairs > to a S3 repository (producint 850 snapshots)repository. > - every day, the one "reading only" cluster of the first point restores > those 850 snapshots to "update" its 1.7k indices from that same S3 > repository. > > It works like a real charm. Load has dropped dramatically, and we can set > a "farm" of temporary machines to do the indexing duties. > > But memory consumption never stops growing. > > we don't get any "out of memory" error or anything. In fact, there is > nothing in the logs that shows any error, but after a week or a few days, > the host has its memory almost exhausted and elasticsearch is not > responding. The memory consumption is of course way ahead of the HEAP_SIZE > We have to restart it and, when we do it we get the following error: > > java.util.concurrent.RejectedExecutionException: Worker has already been > shutdown > at org.elasticsearch.common.netty.channel.socket.nio. > AbstractNioSelector.registerTask(AbstractNioSelector.java:120) > at org.elasticsearch.common.netty.channel.socket.nio. > AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72) > at org.elasticsearch.common.netty.channel.socket.nio.NioWorker. > executeInIoThread(NioWorker.java:36) > at org.elasticsearch.common.netty.channel.socket.nio. > AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56) > at org.elasticsearch.common.netty.channel.socket.nio.NioWorker. > executeInIoThread(NioWorker.java:36) > at org.elasticsearch.common.netty.channel.socket.nio. > AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > execute(DefaultChannelPipeline.java:636) > at org.elasticsearch.common.netty.channel.Channels. > fireExceptionCaughtLater(Channels.java:496) > at org.elasticsearch.common.netty.channel.AbstractChannelSink. > exceptionCaught(AbstractChannelSink.java:46) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > notifyHandlerException(DefaultChannelPipeline.java:658) > at org.elasticsearch.common.netty.channel. > DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream( > DefaultChannelPipeline.java:781) > at org.elasticsearch.common.netty.channel.Channels.write(Channels. > java:725) > at org.elasticsearch.common.netty.handler.codec.oneone. > OneToOneEncoder.doEncode(OneToOneEncoder.java:71) > at org.elasticsearch.common.netty.handler.codec.oneone. > OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > sendDownstream(DefaultChannelPipeline.java:591) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > sendDownstream(DefaultChannelPipeline.java:582) > at org.elasticsearch.common.netty.channel.Channels.write(Channels. > java:704) > at org.elasticsearch.common.netty.channel.Channels.write(Channels. > java:671) > at org.elasticsearch.common.netty.channel.AbstractChannel.write( > AbstractChannel.java:248) > at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse( > NettyHttpChannel.java:158) > at org.elasticsearch.rest.action.search.RestSearchAction$1. > onResponse(RestSearchAction.java:106) > at org.elasticsearch.rest.action.search.RestSearchAction$1. > onResponse(RestSearchAction.java:98) > at org.elasticsearch.action.search.type. > TransportSearchQueryAndFetchAction$AsyncAction.innerFinishHim( > TransportSearchQueryAndFetchAction.java:94) > at org.elasticsearch.action.search.type. > TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase( > TransportSearchQueryAndFetchAction.java:77) > at org.elasticsearch.action.search.type. > TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase( > TransportSearchTypeAction.java:425) > at org.elasticsearch.action.search.type. > TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult( > TransportSearchTypeAction.java:243) > at org.elasticsearch.action.search.<span style="color: # > ... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7374c233-51b4-4697-b534-2da65ddfb967%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
