Excellent, I'm going out of office. I will rerun my test with latest version of master tomorrow.
On Tue, Sep 25, 2018 at 3:03 PM Andrew 👽 Yourtchenko <ayour...@gmail.com> wrote: > Hmm this output does look exactly like what I had before commit ed435504 - > the IPv6 table bucket area gets corrupted by the overruns from IPv4 table > arena. > > And then I spotted the leak which was happening during the rehash which I > fixed in 11521387. > > Your (very few) freelists look like you don’t have that fix either - > normally you would see something > like in http://paste.ubuntu.com/p/y43MsrdHSr/ - notice how many short > elements are in the freelists. > > And given your tests confirm the session cleaner optimization work as I > intended, I have it now in master. > > So you can just pull your fresh tree from master and recheck :) > > --a > > On 25 Sep 2018, at 13:10, khers <s3m2e1.6s...@gmail.com> wrote: > > I checked out from gerrrit, I think it's using latest master. ( but i > make another version to make sure ) > let me explain how I produce this situation, > > while true :) > 1- run trex > ./t-rex-64 --cfg cfg/trex_config.yaml -f cap2/sfr.yaml -m 7 -c 2 > 2-stop trex after ~120 seconds > 3- wait until all session deleted from 'sh acl-plugin session' > > As you see I waited until all session is deleted so all bucket must be > completely free, > I send you part of 'show acl-plugin sessions verbose 1' in this link > <https://paste.ubuntu.com/p/QX4vnGSw7P/>. > > > > On Tue, Sep 25, 2018 at 1:52 PM Andrew 👽 Yourtchenko <ayour...@gmail.com> > wrote: > >> Are you using latest master ? I fixed a couple of issues in bihash last >> week related to memory usage... if it’s the latest master, the output of >> used vs available looks weird... - so please let me know... >> >> As for the “general” growth - basically what happens is bihash doubles >> each bucket size whenever there is a collision on insert, and then converts >> the bucket into linear lookup whenever there is still a collision after >> that growth. >> >> Then the only time the shrinkage/reset is happening is when the bucket is >> completely free - which with long living sessions with overlapping >> lifetimes might mean never. >> >> So one approach to this is to increase the number of buckets. Then they >> will be smaller and have higher probability of being freed. >> >> This is assuming there is nothing else “funny” going on. You can do “show >> acl-plugin sessions verbose 1” via vppctl (It will take forever to complete >> and needs pager disabled since it dumps the entire bihash) to inspect the >> way the buckets are filled... >> >> --a >> >> On 25 Sep 2018, at 12:12, khers <s3m2e1.6s...@gmail.com> wrote: >> >> It's amazing!!! >> >> IPv4 Session lookup hash table: >> Hash table ACL plugin FA IPv4 session bihash >> 968086 active elements 65536 active buckets >> 13 free lists >> [len 16] 1 free elts >> [len 32] 1 free elts >> [len 256] 10669 free elts >> [len 512] 36768 free elts >> [len 1024] 4110 free elts >> [len 2048] 156 free elts >> [len 4096] 4 free elts >> 844 linear search buckets >> arena: base 7fe912320000, next 2680ca780 >> * used 10335594368 b (9856 Mbytes) of 10000000000 b (9536 >> Mbytes)* >> >> >> On Tue, Sep 25, 2018 at 1:39 PM khers <s3m2e1.6s...@gmail.com> wrote: >> >>> Yes, that's right. I think is completely another issue from the patch >>> you sent >>> >>> On Tue, Sep 25, 2018 at 1:35 PM Andrew 👽 Yourtchenko < >>> ayour...@gmail.com> wrote: >>> >>>> Excellent, thanks! >>>> >>>> Memory usage - you mean in bihash arena ? >>>> >>>> --a >>>> >>>> On 25 Sep 2018, at 11:38, khers <s3m2e1.6s...@gmail.com> wrote: >>>> >>>> Throughput and session add/del is stable as rock. The only danger i see >>>> is growing memory usage. >>>> look at this <https://paste.ubuntu.com/p/Hrv6gxzJQT/> >>>> >>>> >>>> On Tue, Sep 25, 2018 at 11:31 AM khers <s3m2e1.6s...@gmail.com> wrote: >>>> >>>>> Of course, I test your patch, there is no slowdown with my scenario. I >>>>> need more time to test other >>>>> scenarios and make sure. >>>>> >>>>> >>>>> On Mon, Sep 24, 2018 at 3:11 PM Andrew 👽 Yourtchenko < >>>>> ayour...@gmail.com> wrote: >>>>> >>>>>> Cool. Then it is probably indeed the session requeues that are not >>>>>> yet efficient... I have been looking at optimizing that. >>>>>> >>>>>> I have a draft in the works which should have less session requeues - >>>>>> I have just added you to it, could you give it a shot and see if it makes >>>>>> things better ? >>>>>> >>>>>> --a >>>>>> >>>>>> On 24 Sep 2018, at 12:55, khers <s3m2e1.6s...@gmail.com> wrote: >>>>>> >>>>>> yes, I confirm >>>>>> >>>>>> On Mon, Sep 24, 2018 at 2:08 PM Andrew 👽 Yourtchenko < >>>>>> ayour...@gmail.com> wrote: >>>>>> >>>>>>> Okay, so what I think I am hearing - the gradual slowdown is/was >>>>>>> always there, and is somewhat more pronounced in master, right ? >>>>>>> >>>>>>> --a >>>>>>> >>>>>>> On 24 Sep 2018, at 11:49, khers <s3m2e1.6s...@gmail.com> wrote: >>>>>>> >>>>>>> I allways get SIGSEGV or 'worker thread dead lock' In 1804 with 1 or >>>>>>> more worker thread and 1 main, >>>>>>> but when vpp using one cpu I hadn't any problem. In the 1807 multi >>>>>>> core is stable i didn't see any of those >>>>>>> problem but throughput is declining slowly. >>>>>>> I ran another test with same version of last email, which vpp is >>>>>>> configured with one core and throughput is declining slower than >>>>>>> master >>>>>>> second 200 <https://paste.ubuntu.com/p/q2MwYX9PRt/> >>>>>>> second 5900 <https://paste.ubuntu.com/p/6ZCDJvB5pg/> >>>>>>> >>>>>>> >>>>>>> On Sun, Sep 23, 2018 at 6:57 PM Andrew 👽 Yourtchenko < >>>>>>> ayour...@gmail.com> wrote: >>>>>>> >>>>>>>> Interesting - but you are saying in 1804 this effect is not >>>>>>>> observed ? There was no other notable changes with regards to session >>>>>>>> management - but maybe worth it to just do hit bisect and see. Should >>>>>>>> be >>>>>>>> 4-5 iterations. Could you verify that - if indeed this is not seen in >>>>>>>> 1804. >>>>>>>> >>>>>>>> --a >>>>>>>> >>>>>>>> On 23 Sep 2018, at 16:42, khers <s3m2e1.6s...@gmail.com> wrote: >>>>>>>> >>>>>>>> I checked out the version before the gerrit 12770 is merged to >>>>>>>> master. >>>>>>>> 2371c25fed6b2e751163df590bb9d9a93a75a0f >>>>>>>> >>>>>>>> I got SIGSEGV with 2 workers, so I repeat the test with one worker. >>>>>>>> Throughput is going down like the latest version. >>>>>>>> >>>>>>>> On Sun, Sep 23, 2018 at 4:55 PM Andrew 👽 Yourtchenko < >>>>>>>> ayour...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Would you be able to confirm that it changes at a point of >>>>>>>>> https://gerrit.fd.io/r/#/c/12770/ ? >>>>>>>>> >>>>>>>>> --a >>>>>>>>> >>>>>>>>> On 23 Sep 2018, at 13:31, emma sdi <s3m2e1.6s...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Dear Community >>>>>>>>> >>>>>>>>> I have simple configuration as following: >>>>>>>>> >>>>>>>>> startup.conf <https://paste.ubuntu.com/p/bbsR2f6k47/> >>>>>>>>> simple_acl <https://paste.ubuntu.com/p/YRM3d77k84/> >>>>>>>>> >>>>>>>>> I used Trex packet generator with following command: >>>>>>>>> ./t-rex-64 --cfg cfg/trex_config.yaml -f cap2/sfr.yaml -m 5 -c 2 >>>>>>>>> -d 6000 >>>>>>>>> The Total-RX gradually decrease, here is output of Trex in second >>>>>>>>> 200 <https://paste.ubuntu.com/p/WJshBskTf5/>, and 5900. >>>>>>>>> <https://paste.ubuntu.com/p/tZHjcPD2rN/> >>>>>>>>> >>>>>>>>> I did not saw this problem in 18.04. I think session_cleaner >>>>>>>>> thread make so many >>>>>>>>> interrupt, do you have any idea? >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>>>>>> Links: You receive all messages sent to this group. >>>>>>>>> >>>>>>>>> View/Reply Online (#10615): >>>>>>>>> https://lists.fd.io/g/vpp-dev/message/10615 >>>>>>>>> Mute This Topic: https://lists.fd.io/mt/26145401/675608 >>>>>>>>> Group Owner: vpp-dev+ow...@lists.fd.io >>>>>>>>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [ >>>>>>>>> ayour...@gmail.com] >>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=- >>>>>>>>> >>>>>>>>>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10653): https://lists.fd.io/g/vpp-dev/message/10653 Mute This Topic: https://lists.fd.io/mt/26145401/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
