Re: Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

[Joel Bernstein]

Yes the hashJoin will read the entire "hashed" query into memory. The
documentation explains this.

In general the streaming joins were designed for OLAP type work loads.
Unless you have a large cluster powering streaming joins you are going to
have problems with high QPS workloads.

Joel Bernstein
http://joelsolr.blogspot.com/

On Sun, Jul 9, 2017 at 10:59 PM, Zheng Lin Edwin Yeo 
wrote:

> I have found that it could be likely due to the hashJoin in the streaming
> expression, as this will store all tuples in memory?
>
> I have more than 12 million in the collections which I am querying, in 1
> shard. The index size of the collection is 45 GB.
> Physical RAM of server: 384 GB
> Java Heap: 22 GB
> Typical search latency: 2 to 4 seconds
>
> Regards,
> Edwin
>
>
> On 7 July 2017 at 16:46, Jan Høydahl  wrote:
>
> > You have not told us how many documents you have, how many shards, how
> big
> > the docs are, physical RAM, Java heap, what typical search latency is
> etc.
> >
> > If you have tried to squeeze too many docs into a single node it might
> get
> > overloaded faster, thus sharding would help.
> > If you return too much content (large fields that you won’t use) that may
> > lower the max QPS for a node, so check that.
> > If you are not using DocValues, faceting etc will take too much memory,
> > but since you use streaming I guess you use Docvalues.
> > There are products that you can put in front of Solr that can do rate
> > limiting for you, such as https://getkong.org/ 
> >
> > You really need to debug what is the bottleneck in your case and try to
> > fix that.
> >
> > Can you share your key numbers here so we can do a qualified guess?
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> >
> > > 2. jul. 2017 kl. 09.00 skrev Zheng Lin Edwin Yeo  >:
> > >
> > > Hi,
> > >
> > > I'm currently facing the issue whereby the Solr crashed when I have
> > issued
> > > too many queries with error or those with high memory usage, like JSON
> > > facet or Streaming expressions.
> > >
> > > What could be the issue here?
> > >
> > > I'm using Solr 6.5.1
> > >
> > > Regards,
> > > Edwin
> >
> >
>

Re: Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

[Zheng Lin Edwin Yeo]

I have found that it could be likely due to the hashJoin in the streaming
expression, as this will store all tuples in memory?

I have more than 12 million in the collections which I am querying, in 1
shard. The index size of the collection is 45 GB.
Physical RAM of server: 384 GB
Java Heap: 22 GB
Typical search latency: 2 to 4 seconds

Regards,
Edwin


On 7 July 2017 at 16:46, Jan Høydahl  wrote:

> You have not told us how many documents you have, how many shards, how big
> the docs are, physical RAM, Java heap, what typical search latency is etc.
>
> If you have tried to squeeze too many docs into a single node it might get
> overloaded faster, thus sharding would help.
> If you return too much content (large fields that you won’t use) that may
> lower the max QPS for a node, so check that.
> If you are not using DocValues, faceting etc will take too much memory,
> but since you use streaming I guess you use Docvalues.
> There are products that you can put in front of Solr that can do rate
> limiting for you, such as https://getkong.org/ 
>
> You really need to debug what is the bottleneck in your case and try to
> fix that.
>
> Can you share your key numbers here so we can do a qualified guess?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 2. jul. 2017 kl. 09.00 skrev Zheng Lin Edwin Yeo :
> >
> > Hi,
> >
> > I'm currently facing the issue whereby the Solr crashed when I have
> issued
> > too many queries with error or those with high memory usage, like JSON
> > facet or Streaming expressions.
> >
> > What could be the issue here?
> >
> > I'm using Solr 6.5.1
> >
> > Regards,
> > Edwin
>
>

Re: Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

[Jan Høydahl]

You have not told us how many documents you have, how many shards, how big the 
docs are, physical RAM, Java heap, what typical search latency is etc.

If you have tried to squeeze too many docs into a single node it might get 
overloaded faster, thus sharding would help.
If you return too much content (large fields that you won’t use) that may lower 
the max QPS for a node, so check that.
If you are not using DocValues, faceting etc will take too much memory, but 
since you use streaming I guess you use Docvalues.
There are products that you can put in front of Solr that can do rate limiting 
for you, such as https://getkong.org/ 

You really need to debug what is the bottleneck in your case and try to fix 
that.

Can you share your key numbers here so we can do a qualified guess?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 2. jul. 2017 kl. 09.00 skrev Zheng Lin Edwin Yeo :
> 
> Hi,
> 
> I'm currently facing the issue whereby the Solr crashed when I have issued
> too many queries with error or those with high memory usage, like JSON
> facet or Streaming expressions.
> 
> What could be the issue here?
> 
> I'm using Solr 6.5.1
> 
> Regards,
> Edwin

Re: Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

[Toke Eskildsen]

On Sun, 2017-07-02 at 15:00 +0800, Zheng Lin Edwin Yeo wrote:
> I'm currently facing the issue whereby the Solr crashed when I have
> issued too many queries with error or those with high memory usage,
> like JSON facet or Streaming expressions.
> 
> What could be the issue here?

Solr does not have any auto-limiting of the number of concurrent
requests. You will have to build that yourself (quite hard) or impose a
hard limit in your request layer that is low enough to guarantee that
you don't run out of memory in Solr.

You could raise the amount of memory allocated for Solr, but even then
you might want to have a hard limit, just to avoid the occasional "cat
steps on F5 and the browser issues a gazillion requests"-scenario.
-- 
Toke Eskildsen, Royal Danish Library

Re: Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

[Rick Leir]

Stack trace? Memory diagnostics from top(1)? What querys?

On July 2, 2017 3:00:16 AM EDT, Zheng Lin Edwin Yeo  
wrote:
>Hi,
>
>I'm currently facing the issue whereby the Solr crashed when I have
>issued
>too many queries with error or those with high memory usage, like JSON
>facet or Streaming expressions.
>
>What could be the issue here?
>
>I'm using Solr 6.5.1
>
>Regards,
>Edwin

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com
-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

[Zheng Lin Edwin Yeo]

Hi,

I'm currently facing the issue whereby the Solr crashed when I have issued
too many queries with error or those with high memory usage, like JSON
facet or Streaming expressions.

What could be the issue here?

I'm using Solr 6.5.1

Regards,
Edwin