Re: 6.6 cloud starting to eat CPU after 8+ hours

; Puzzling.. > > Thanks, > Markus > > -Original message- > > From:Mikhail Khludnev <m...@apache.org> > > Sent: Wednesday 26th July 2017 10:50 > > To: solr-user <solr-user@lucene.apache.org> > > Subject: Re: 6.6 cloud starting to eat CPU

RE: 6.6 cloud starting to eat CPU after 8+ hours

gt; > > > > > > -Original message----- > > > From:Mikhail Khludnev <m...@apache.org> > > > Sent: Wednesday 19th July 2017 14:41 > > > To: solr-user <solr-user@lucene.apache.org> > > > Subject: Re: 6.6 cloud starting to eat CPU after 8

Re: 6.6 cloud starting to eat CPU after 8+ hours

ts are being > indexed. Some queries are executed but not very much. Attaching the stack > anyway. > > > > > > -Original message- > > From:Mikhail Khludnev <m...@apache.org> > > Sent: Wednesday 19th July 2017 14:41 > > To: solr-user <sol

RE: 6.6 cloud starting to eat CPU after 8+ hours

: Markus Jelsma <markus.jel...@openindex.io> > Subject: Re: 6.6 cloud starting to eat CPU after 8+ hours > > On 7/19/2017 3:35 AM, Markus Jelsma wrote: > > Another peculiarity here, our six node (2 shards / 3 replica's) cluster is > > going crazy after a good part o

Re: 6.6 cloud starting to eat CPU after 8+ hours

nt: Wednesday 19th July 2017 14:41 >> To: solr-user <solr-user@lucene.apache.org> >> Subject: Re: 6.6 cloud starting to eat CPU after 8+ hours >> >> You can get stack from kill -3 jstack even from solradmin. Overall, this >> behavior looks like typical heavy merge ki

RE: 6.6 cloud starting to eat CPU after 8+ hours

? Thanks, Markus -Original message- > From:Mikhail Khludnev <m...@apache.org> > Sent: Wednesday 19th July 2017 14:41 > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: 6.6 cloud starting to eat CPU after 8+ hours > > You can get stack from kill -3 jstac

Re: 6.6 cloud starting to eat CPU after 8+ hours

ly start to go > bad, until nodes are restarted (or index size decreased). > > Thanks, > Markus > > -Original message- > > From:Mikhail Khludnev <m...@apache.org> > > Sent: Wednesday 19th July 2017 14:18 > > To: solr-user <solr-user@lu

RE: 6.6 cloud starting to eat CPU after 8+ hours

: Re: 6.6 cloud starting to eat CPU after 8+ hours > > > > > The real distinction between busy and calm nodes is that busy nodes all > > have o.a.l.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms() as > > second to fillBuffer(), what are they doing? >

Re: 6.6 cloud starting to eat CPU after 8+ hours

> > The real distinction between busy and calm nodes is that busy nodes all > have o.a.l.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms() as > second to fillBuffer(), what are they doing? Can you expose the stack deeper? Can they start to sync shards due to some reason? On Wed, Jul

RE: 6.6 cloud starting to eat CPU after 8+ hours

These are all SSD's. Thanks, Markus -Original message- > From:Rick Leir <rl...@leirtech.com> > Sent: Wednesday 19th July 2017 12:48 > To: solr-user@lucene.apache.org > Subject: Re: 6.6 cloud starting to eat CPU after 8+ hours > > Markus, > What does iostat(1) tell you?

Re: 6.6 cloud starting to eat CPU after 8+ hours

Markus, What does iostat(1) tell you? Cheers -- Rick On July 19, 2017 5:35:32 AM EDT, Markus Jelsma wrote: >Hello, > >Another peculiarity here, our six node (2 shards / 3 replica's) cluster >is going crazy after a good part of the day has passed. It starts >eating

6.6 cloud starting to eat CPU after 8+ hours

Hello, Another peculiarity here, our six node (2 shards / 3 replica's) cluster is going crazy after a good part of the day has passed. It starts eating CPU for no good reason and its latency goes up. Grafana graphs show the problem really well After restarting 2/6 nodes, there is also quite a