Yes the hashJoin will read the entire "hashed" query into memory. The documentation explains this.
In general the streaming joins were designed for OLAP type work loads. Unless you have a large cluster powering streaming joins you are going to have problems with high QPS workloads. Joel Bernstein http://joelsolr.blogspot.com/ On Sun, Jul 9, 2017 at 10:59 PM, Zheng Lin Edwin Yeo <edwinye...@gmail.com> wrote: > I have found that it could be likely due to the hashJoin in the streaming > expression, as this will store all tuples in memory? > > I have more than 12 million in the collections which I am querying, in 1 > shard. The index size of the collection is 45 GB. > Physical RAM of server: 384 GB > Java Heap: 22 GB > Typical search latency: 2 to 4 seconds > > Regards, > Edwin > > > On 7 July 2017 at 16:46, Jan Høydahl <jan....@cominvent.com> wrote: > > > You have not told us how many documents you have, how many shards, how > big > > the docs are, physical RAM, Java heap, what typical search latency is > etc. > > > > If you have tried to squeeze too many docs into a single node it might > get > > overloaded faster, thus sharding would help. > > If you return too much content (large fields that you won’t use) that may > > lower the max QPS for a node, so check that. > > If you are not using DocValues, faceting etc will take too much memory, > > but since you use streaming I guess you use Docvalues. > > There are products that you can put in front of Solr that can do rate > > limiting for you, such as https://getkong.org/ <https://getkong.org/> > > > > You really need to debug what is the bottleneck in your case and try to > > fix that. > > > > Can you share your key numbers here so we can do a qualified guess? > > > > -- > > Jan Høydahl, search solution architect > > Cominvent AS - www.cominvent.com > > > > > 2. jul. 2017 kl. 09.00 skrev Zheng Lin Edwin Yeo <edwinye...@gmail.com > >: > > > > > > Hi, > > > > > > I'm currently facing the issue whereby the Solr crashed when I have > > issued > > > too many queries with error or those with high memory usage, like JSON > > > facet or Streaming expressions. > > > > > > What could be the issue here? > > > > > > I'm using Solr 6.5.1 > > > > > > Regards, > > > Edwin > > > > >
