You can specify several selectors, 'class' and 'labels' are lists, see definition of DirectoryRequest thrift structure.
Can you share the number of comparisons which were performed for each request? On Friday, May 29, 2020 at 4:34:56 PM UTC+2, A. Hébert wrote: > > I run this test and I have the following result: > > - Retrieving 1 series takes 6269551620 ns (6 s) > - Retrieving all 14 series takes 197156864266 ns (3m 17s), I had to use a > regular expression as I couldn't specified several classes (may be that's > why it's a bit slower that previous find queries). > > However I didn't get 1 or 14 series as result but 3 and 42. > > This test leads me to think that the bottleneck is related to the directory > component or at least our directories architecture. In production we have: > - 6 directories > - 3 replicas > - 2 shards per replica > > - Between 210 and 230 millions of series per directory > > Le lundi 18 mai 2020 14:47:22 UTC+2, Mathias Herberts a écrit : >> >> You can try fetching directly from one of your Directory using the class >> at https://gist.github.com/hbs/30bf6921b5145e0474adae3273027ef3 >> >> It is called like this: >> >> java -cp warp10/build/libs/warp10-x.y.z.jar:. DirectoryStreamClient "{ >> 'urls' [ 'http://HOST:PORT/directory-streaming' ] 'psk' 'HEX ENCODED >> PRESHARED KEY' 'class' [ 'CLASS_SELECTOR' ] 'labels' [ { /* LABEL SELECTOR >> */ } ] }" >> >> >> It will output the list of fetched GTS (their Metadata) and the time >> taken for the request in ns. >> >> >> This should help you determine if the Directory instance itself is the >> bottleneck of something else down the line. >> >> On Monday, May 18, 2020 at 9:42:50 AM UTC+2, A. Hébert wrote: >>> >>> We do have fast queries on our indexes too (on average our find queries >>> took 150 to 200ms). Our directories are not too overloaded (between 40 and >>> 60 % of cpu load). In our use-cases we do have specific slow queries on >>> high class names cardinalities. For instance if 20/30 accounts pushed the >>> same class more than 1 millions times (ex: my.cpu, my.mem, ...). Then even >>> a new account pushing this specific series is impacted by slow index >>> queries too. >>> >>> I was more searching for the root cause of this behaviour / an hint on >>> how to improve those data retrieval. >>> >>> For information with the /find request it took the same time (real: >>> 1m13.770s) >>> >>> Le lundi 18 mai 2020 08:56:07 UTC+2, Mathias Herberts a écrit : >>>> >>>> 1.3M comparisons on 435k metadata usually does not take 25s, this is a >>>> typical log in one of our own systems: >>>> >>>> Search returned 27 results in 183.488054 ms, inspected 182304 metadatas >>>> in 1 classes (1 matched) and performed 546912 comparisons >>>> >>>> >>>> what is the load average on your Directory? What is the time a /find >>>> request actually takes (the path is slightly different so result may give >>>> some additional infos). >>>> >>>> >>>> >>>> On Friday, May 15, 2020 at 5:35:21 PM UTC+2, A. Hébert wrote: >>>>> >>>>> I can't really provides those lines with certainty... We are using HA >>>>> sharded directories. I could find an example of what may like a slow >>>>> query: >>>>> Search returned 14 results in 25218.834073 ms, inspected 435067 >>>>> metadatas in 1 classes (1 matched) and performed 1306154 comparisons. >>>>> >>>>> For information in all the sharded directory logs, I didn't found a >>>>> log with 14 results [ ... ] in 14 classes or with 14 matched neither. >>>>> >>>>> Le vendredi 15 mai 2020 09:45:12 UTC+2, Mathias Herberts a écrit : >>>>>> >>>>>> Can you provide the directory log lines related to the searches you >>>>>> are performing? They should look like: >>>>>> >>>>>> Search returned AAA results in BBB ms, inspected CCC metadatas in DDD >>>>>> classes (EEE matched) and performed FFF comparisons >>>>>> >>>>>> >>>>>> >>>>>> On Thursday, May 14, 2020 at 4:53:33 PM UTC+2, A. Hébert wrote: >>>>>>> >>>>>>> I do apologise, I run my tests to fast, the 'extra' parameter isn't >>>>>>> working with a FIND statement only with FETCH. I couldn't really reduce >>>>>>> by >>>>>>> a lot using this method (from 70 - 80 seconds) to (40 - 50 seconds) to >>>>>>> retrieve the 14 last values of those series. I had to split my >>>>>>> statement in >>>>>>> 3 or 4 FETCH, as the labels keys are different for several classes. >>>>>>> >>>>>>> Le jeudi 14 mai 2020 16:42:02 UTC+2, A. Hébert a écrit : >>>>>>>> >>>>>>>> Thanks for your proposal. Do you have an example for the 'extra' >>>>>>>> parameter? I couldn't make it work (only one series as result) on a >>>>>>>> Warp10 >>>>>>>> 2.5. >>>>>>>> As a matter of fact this time I am really in the second case (I >>>>>>>> can't know the full set of labels only the id) and I really would like >>>>>>>> to >>>>>>>> be able to reduce the 10 seconds to FIND a single series. >>>>>>>> >>>>>>>> Le mardi 28 avril 2020 21:34:03 UTC+2, Mathias Herberts a écrit : >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> assuming you know the classes of those 14 series you want to >>>>>>>>> access, you could very well use the same approach using the 'gts' >>>>>>>>> parameter >>>>>>>>> with a list of GTS you build manually, i.e.: >>>>>>>>> >>>>>>>>> 'gts' [ >>>>>>>>> NEWGTS 'class-1' RENAME { 'id' '42' } RELABEL >>>>>>>>> NEWGTS 'class-2' RENAME { 'id' '42' } RELABEL >>>>>>>>> .. >>>>>>>>> ] >>>>>>>>> >>>>>>>>> Of course you need to specify the full set of labels so this >>>>>>>>> approach will only work if you know what the labels are. >>>>>>>>> >>>>>>>>> Another approach, assuming you do not know what the labels are but >>>>>>>>> you know the class names, you could select the GTS using a single >>>>>>>>> class >>>>>>>>> name then use the 'extra' parameter to specify the other 13 classes. >>>>>>>>> >>>>>>>>> On Monday, April 27, 2020 at 11:54:41 AM UTC+2, A. Hébert wrote: >>>>>>>>>> >>>>>>>>>> Hello Mathias, >>>>>>>>>> >>>>>>>>>> I am back again on this story of performance for queries due to >>>>>>>>>> the Index size. >>>>>>>>>> >>>>>>>>>> First of all, I wanted to say that in many use-cases the FETCH >>>>>>>>>> with the "gts" key improve a lot of query in our side. >>>>>>>>>> >>>>>>>>>> However we still have massive slow queries, for example when we >>>>>>>>>> want to compute a FIND on a specific label on one of our most large >>>>>>>>>> application. The selector query is '~.*{id=42}' and this queries >>>>>>>>>> takes more >>>>>>>>>> than 70 seconds (server side). The FINDSTATS provides the following >>>>>>>>>> result >>>>>>>>>> "{"classes.estimate":14,"gts.estimate":14,"labelnames.estimate":13,"labelvalues.estimate":12,"error.rate":0.008125}". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Even when setting one series name on the selector 'test{id=42}', >>>>>>>>>> we still get series than takes more than 10 seconds to be executed >>>>>>>>>> (server >>>>>>>>>> side). >>>>>>>>>> >>>>>>>>>> I would be to help to improve those queries. What can I look like >>>>>>>>>> to start with? >>>>>>>>>> >>>>>>>>>> Le mercredi 11 mars 2020 17:01:50 UTC+1, A. Hébert a écrit : >>>>>>>>>>> >>>>>>>>>>> For sure on a batch of hundred queries an average of 150ms, the >>>>>>>>>>> best resolved in only 28ms, and the worst in 1s >>>>>>>>>>> >>>>>>>>>>> Le mercredi 11 mars 2020 16:52:34 UTC+1, Mathias Herberts a >>>>>>>>>>> écrit : >>>>>>>>>>>> >>>>>>>>>>>> Did you see an improvement in terms of performance? >>>>>>>>>>>> >>>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Warp 10 users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/19f58764-8d0b-4d6b-8403-090abc426833%40googlegroups.com.
