Apologies 28 cores each (not 8) On Wed 16 Jan 2019 at 22:11, Olivier Mallassi <[email protected]> wrote:
> All > > I do not have all the details here but > - hardware is large (4 machines, 8 cores each). I do not have cpu metrics > - it appears to be related to regexp evaluation. We tried “register by > key” w/ interestresultpolicy.keys and get better results > > Now > 1/ we are consuming too much memory. If I do the math I am consuming 90MB > per subscription (no ha..) knowing I have 1000 subscriptions it meaans 70GB > if I am not mistaken > A priori the object size is small meaning that I should have a large > number of keys in the subscription queue (that I do not understand) Am I > right? > > 2/ when we start all the client processes (1000) it looks like we need 20 > minutes to have all subscriptions created. Any ideas of what we are missing? > > Any Help is welcomed, Even at the OS level. :) > > Regards > Oliv/ > > > OSat 12 Jan 2019 at 11:05, Nabarun Nag <[email protected]> wrote: > >> Hi, >> >> Could you please share more details about the objects and the hardware >> specs. >> Also, it will be really convenient if you share you git repo. >> Regards >> Naba >> >> On Sat, Jan 12, 2019 at 1:04 AM Olivier Mallassi < >> [email protected]> wrote: >> >>> Anil, Mike >>> >>> Thanks for your answers.AFAIK it is not correlated to GC. I will try to >>> look in more details to the gfs metrics. >>> >>> Oliv/ >>> >>> >>> On Fri 11 Jan 2019 at 20:15, Michael Stolz <[email protected]> wrote: >>> >>>> Olivier, >>>> >>>> Can you correlate anything like GC pauses with these slow deliveries? >>>> >>>> -- >>>> Mike Stolz >>>> Principal Engineer, GemFire Product Lead >>>> Mobile: +1-631-835-4771 >>>> >>>> >>>> >>>> On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <[email protected]> >>>> wrote: >>>> >>>>> Oliv, >>>>> >>>>> The subscription has cost associated with it; as interested need to be >>>>> processed for all the clients; but its minimal looking for matching >>>>> keys/regex. >>>>> - The puts add the matched events to the client subscription queues on >>>>> the server; do you see the client queues growing? >>>>> - You may also need to see if there are any memory pressure on server. >>>>> - If the clients are not consuming the events fast (slow clients) it >>>>> may increase the server side queue size. >>>>> >>>>> -Anil. >>>>> >>>>> >>>>> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi < >>>>> [email protected]> wrote: >>>>> >>>>>> sorry mistake : throughtput is 10 times less: 5k requests per sec. >>>>>> >>>>>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> All >>>>>>> >>>>>>> I am facing a kind of scaling mystery while using register interest. >>>>>>> >>>>>>> Use-case is basic : >>>>>>> N publishers publishing into a geode cluster w/ M subscribers using >>>>>>> register_interest (regexp). >>>>>>> >>>>>>> in our case, >>>>>>> - M can be up to 1000 process (not threads) >>>>>>> - Publisher throughput is about 50k requests per sec on 200k keys >>>>>>> - Volume is small (around 10GB) >>>>>>> >>>>>>> We observe put (from publisher point of view) latency diverging with >>>>>>> the number of consumers, up to 10 sec (p99) and 5 sec (p50) >>>>>>> AFAIR, the key is evaluated on the put (sync). >>>>>>> >>>>>>> Any ideas , customisation of the configuration you can advice? >>>>>>> >>>>>>> Regards. >>>>>>> >>>>>>> Oliv/ >>>>>>> >>>>>>
