Hi Jan,
Upgrading to 2.x version will take at-least 6-7 months for us to
migrate. About the heap, it is at 70% and no issues with it. No logs are
showing up based on the slow rate, apart from what i have posted. There was
an error on indice which mentioned about not able to calculate the range, i
did the "Recalculate index cycles" which fixed that, but the slow
processing has accumulated about 100,000,000 messages in the journal on the
Master Node.
Still finding what is happening.. :-(
Thanks Hema.
On Friday, July 8, 2016 at 5:40:47 PM UTC+5:30, Hema Kumar wrote:
>
> Hi,
> I am using graylog 1.3.3 with ES 1.7.5, from yesterday we are seeing
> the process buffer filled up on the master node and the outgoing process is
> too slow than normal, I have tried restarting GL and ES but did not fix the
> issue, below are the log warn and errors we see that repeats continuously.
>
> We have 4 graylog server and 7 elasticsearch nodes, Only the Master
> graylog is processing slow and sometimes the 3rd node, rest of the nodes
> are working fine.
>
> Could you please help me on this, i have been breaking my head since
> yesterday.
>
>
> 2016-07-08T01:53:21.355-06:00 WARN [GelfChunkAggregator] Error while
> expiring GELF chunk entries
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.graylog2.inputs.codecs.GelfChunkAggregator$ChunkEvictionTask.run(GelfChunkAggregator.java:288)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-07-08T05:37:47.803-06:00 ERROR [AnyExceptionClassMapper] Unhandled
> exception in REST resource
> org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
> execute phase [fetch], [reduce]
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.onFailure(TransportSearchQueryThenFetchAction.java:159)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:41)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassCastException:
> org.elasticsearch.search.aggregations.bucket.terms.LongTerms$Bucket cannot
> be cast to
> org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket
> at
> org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.compareTerm(StringTerms.java:85)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:87)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:83)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:284)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:270)
> at
> org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:37)
> at
> org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:26)
> at
> org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:225)
> at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:133)
> at
> org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:149)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.reduce(InternalTerms.java:195)
> at
> org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
> at
> org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.reduce(InternalSingleBucketAggregation.java:79)
> at
> org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
> at
> org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:407)
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.doRun(TransportSearchQueryThenFetchAction.java:147)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
>
> 2016-07-08T05:43:30.757-06:00 WARN [DefaultExceptionMonitor] Unexpected
> exception.
> java.nio.channels.ClosedSelectorException
> at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)
> at
> org.apache.mina.transport.socket.nio.NioProcessor.isSelectorEmpty(NioProcessor.java:107)
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1139)
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks, Hema.
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/4f40a9a3-8d16-43e7-80a2-06f854cf707c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
