Re: Effects of insert order on query performance

2016-08-12 Thread Jeff Wartes
Thanks Emir. I’m unfortunately already using a routing key that needs to be at the top level, since I’m collapsing on that field. Adding a sub-key won’t help much if my theory is correct, as even a single shard (distrib=false) showed serious performance degradation, and query latency is the

Re: Effects of insert order on query performance

2016-08-12 Thread Emir Arnautovic
Hi Jeff, I will not comment on your theory (will let that to guys more familiar with Lucene code) but will point to one alternative solution: routing. You can use routing to split documents with different permission to different shards and use composite hash routing to split "A" (and maybe

Effects of insert order on query performance

2016-08-11 Thread Jeff Wartes
This isn’t really a question, although some validation would be nice. It’s more of a warning. Tldr is that the insert order of documents in my collection appears to have had a huge effect on my query speed. I have a very large (sharded) SolrCloud 5.4 index. One aspect of this index is a