Daniel,
Is it worth saying that you have honkin' long queries and there must be a
simpler way? ( I am a big fan of KISS . . Keep It Simple Stupid). I am not
calling you names, just saying that this acronym comes up in just about every
project I work on. It is akin to the Peter Principle, where design complexity
inevitably increases to the breaking point, then I get cranky. And you probably
can tell us a solid reason for having the long queries. Cheers -- Rick
On May 30, 2017 9:22:24 AM EDT, Susheel Kumar <susheel2...@gmail.com> wrote:
>If you are able to draw gc logs in gcviewer when OOM happens, it can
>give
>you idea if it was sudden OOM or heap gets filled over a period of
>time.
>This may help to nail down if any particular query is causing the
>problem
>or something else...
>
>Thanks,
>Susheel
>
>On Sat, May 27, 2017 at 5:36 PM, Daniel Angelov
><dani.b.ange...@gmail.com>
>wrote:
>
>> Thanks for the support so far.
>> I am going to analyze the logs in order to check the frequency of
>such
>> queries. BTW, I have forgot to mention, the soft and the hard commits
>are
>> each 60 sec.
>>
>> BR
>> Daniel
>>
>> Am 27.05.2017 22:57 schrieb "Erik Hatcher" <erik.hatc...@gmail.com>:
>>
>> > Another technique to consider is {!join}. Index the cross ref id
>"sets"
>> > to another core and use a short and sweet join, if there are stable
>sets
>> of
>> > id's.
>> >
>> > Erik
>> >
>> > > On May 27, 2017, at 11:39, Alexandre Rafalovitch
><arafa...@gmail.com>
>> > wrote:
>> > >
>> > > On top of Shawn's analysis, I am also wondering how often those
>FQ
>> > > queries are reused. Because they and the matching documents are
>> > > getting cached, so there might be quite a bit of space taken with
>that
>> > > too.
>> > >
>> > > Regards,
>> > > Alex.
>> > > ----
>> > > http://www.solr-start.com/ - Resources for Solr users, new and
>> > experienced
>> > >
>> > >
>> > >> On 27 May 2017 at 11:32, Shawn Heisey <apa...@elyograg.org>
>wrote:
>> > >>> On 5/27/2017 9:05 AM, Shawn Heisey wrote:
>> > >>>> On 5/27/2017 7:14 AM, Daniel Angelov wrote:
>> > >>>> I would like to ask, what could be the memory/cpu impact, if
>the fq
>> > >>>> parameter in many of the queries is a long string (fq={!terms
>> > >>>> f=...}...,.... ) around 2000000 chars. Most of the queries are
>like:
>> > >>>> "q={!frange l=Timestamp1 u=Timestamp2}... + some others
>criteria".
>> > >>>> This is with SolrCloud 4.1, on 10 hosts, 3 collections,
>summary in
>> > >>>> all collections are around 10000000 docs. The queries are over
>all 3
>> > >>>> collections.
>> > >>
>> > >> Followup after a little more thought:
>> > >>
>> > >> If we assume that the terms in your filter query are a generous
>15
>> > >> characters each (plus a comma), that means there are in the
>ballpark
>> of
>> > >> 125 thousand of them in a two million byte filter query. If
>they're
>> > >> smaller, then there would be more. Considering 56 bytes of
>overhead
>> for
>> > >> each one, there's at least another 7 million bytes of memory for
>> 125000
>> > >> terms when the terms parser divides that filter into multiple
>String
>> > >> objects, plus memory required for the data in each of those
>small
>> > >> strings, which will be just a little bit less than the original
>four
>> > >> million bytes, because it will exclude the commas. A fair
>amount of
>> > >> garbage will probably also be generated in order to parse the
>filter
>> ...
>> > >> and then once the query is done, the 15 megabytes (or more) of
>memory
>> > >> for the strings will also be garbage. This is going to repeat
>for
>> every
>> > >> shard.
>> > >>
>> > >> I haven't even discussed what happens for memory requirements on
>the
>> > >> Lucene frange parser, because I don't have any idea what those
>are,
>> and
>> > >> you didn't describe the function you're using. I also don't
>know how
>> > >> much memory Lucene is going to require in order to execute a
>terms
>> > >> filter with at least 125K terms. I don't imagine it's going to
>be
>> > small.
>> > >>
>> > >> Thanks,
>> > >> Shawn
>> > >>
>> >
>>
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com
