All I do not have all the details here but - hardware is large (4 machines, 8 cores each). I do not have cpu metrics - it appears to be related to regexp evaluation. We tried “register by key” w/ interestresultpolicy.keys and get better results
Now 1/ we are consuming too much memory. If I do the math I am consuming 90MB per subscription (no ha..) knowing I have 1000 subscriptions it meaans 70GB if I am not mistaken A priori the object size is small meaning that I should have a large number of keys in the subscription queue (that I do not understand) Am I right? 2/ when we start all the client processes (1000) it looks like we need 20 minutes to have all subscriptions created. Any ideas of what we are missing? Any Help is welcomed, Even at the OS level. :) Regards Oliv/ OSat 12 Jan 2019 at 11:05, Nabarun Nag <[email protected]> wrote: > Hi, > > Could you please share more details about the objects and the hardware > specs. > Also, it will be really convenient if you share you git repo. > Regards > Naba > > On Sat, Jan 12, 2019 at 1:04 AM Olivier Mallassi < > [email protected]> wrote: > >> Anil, Mike >> >> Thanks for your answers.AFAIK it is not correlated to GC. I will try to >> look in more details to the gfs metrics. >> >> Oliv/ >> >> >> On Fri 11 Jan 2019 at 20:15, Michael Stolz <[email protected]> wrote: >> >>> Olivier, >>> >>> Can you correlate anything like GC pauses with these slow deliveries? >>> >>> -- >>> Mike Stolz >>> Principal Engineer, GemFire Product Lead >>> Mobile: +1-631-835-4771 >>> >>> >>> >>> On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <[email protected]> >>> wrote: >>> >>>> Oliv, >>>> >>>> The subscription has cost associated with it; as interested need to be >>>> processed for all the clients; but its minimal looking for matching >>>> keys/regex. >>>> - The puts add the matched events to the client subscription queues on >>>> the server; do you see the client queues growing? >>>> - You may also need to see if there are any memory pressure on server. >>>> - If the clients are not consuming the events fast (slow clients) it >>>> may increase the server side queue size. >>>> >>>> -Anil. >>>> >>>> >>>> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi < >>>> [email protected]> wrote: >>>> >>>>> sorry mistake : throughtput is 10 times less: 5k requests per sec. >>>>> >>>>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi < >>>>> [email protected]> wrote: >>>>> >>>>>> All >>>>>> >>>>>> I am facing a kind of scaling mystery while using register interest. >>>>>> >>>>>> Use-case is basic : >>>>>> N publishers publishing into a geode cluster w/ M subscribers using >>>>>> register_interest (regexp). >>>>>> >>>>>> in our case, >>>>>> - M can be up to 1000 process (not threads) >>>>>> - Publisher throughput is about 50k requests per sec on 200k keys >>>>>> - Volume is small (around 10GB) >>>>>> >>>>>> We observe put (from publisher point of view) latency diverging with >>>>>> the number of consumers, up to 10 sec (p99) and 5 sec (p50) >>>>>> AFAIR, the key is evaluated on the put (sync). >>>>>> >>>>>> Any ideas , customisation of the configuration you can advice? >>>>>> >>>>>> Regards. >>>>>> >>>>>> Oliv/ >>>>>> >>>>>
