Re: [vpp-dev] 答复: 【vpp-dev】delay is error in ping with multi worker thread
Cool, also would be glad to hear from the community :p zhang...@yunshan.net.cn 发件人： 张东亚 发送时间： 2017-06-22 13:30 收件人： zhang...@yunshan.net.cn 抄送： 雷彦章; vpp-dev 主题： Re: Re: [vpp-dev]答复: 【vpp-dev】delay is error in ping with multi worker thread Hi zhangpan, Glad to here feedback, as I written in my previous mail, after we modify the code to using RPC mechanism, ping works fine but we have random 10ms issue while our normal ping delay is <1ms, we think the reason it due to schedule delay in the main thread but we want the community to confirm that. For vector processing, I think there is a upper limit for wait packets in order to reduce packet processing delay, it should not wait that much for a batch. 2017-06-22 13:11 GMT+08:00 zhang...@yunshan.net.cn <zhang...@yunshan.net.cn>: I didn't investigate the code but there's an assumption I would like to share. I encountered the same problem before, but what interested me is that I found if you send multiple ping pkts as fast as you can, the problem went away. I tried different numbers and if it's a multiple of 4, VPP works just as expected. I just assumed that in order to perform “vector processing”, VPP first waits for enough( > 4) pkts to store in a vector and then, process the batch. If you send ICMP pkt one by one, VPP will only start to process these pkts after receiving at lease 4 pkts. So for the first ping, the latency is 1000ms. Best Regards, Pan zhang...@yunshan.net.cn 发件人： 张东亚 发送时间： 2017-06-22 11:09 收件人： 雷彦章 抄送： vpp-dev 主题： Re: [vpp-dev]答复: 【vpp-dev】delay is error in ping with multi worker thread Hi List, I am colleague of Leiyanzhang, let me describe share the issues and investigations found when we test the ping command under multiple worker thread case. Stage1: Issue: Ping does not work under multiple worker thread case. Version: Vanilla vpp 17.04 (vanilla vp) Stage2: Issue: Ping got > 1s delay result under multiple worker thread case, 1s is the default ping interval, if we specify 10s interval, the delay will be 10s delay. Version: Vanilla vpp 17.04 + VPP-844 patch. Our Solution: as Lei pointed out, we think it's not correct for the worker thread to directly using the ping_run_t's cli_process_id to notify the main thread because the vlib_process_signal_event is using work thread vlib_main structure, however since process is shared by main thread and worker, when cli process in main thread timeout waiting event, it can still fetch the event, but the delay will be the interval. State3: Issue: Ping delay is not related with interval, but got 10ms+ delay randomly. Version: Vanilla vpp 17.04 + VPP-844 Patch + change to using vl_api_rpc_call_main_thread to notify cli process in main thread. Our solution: After read vpp code, we think worker thread should use vl_api_rpc_call_main_thread to notify cli process in main thread, after will made the fix, the ping command works as expected most time with only random 10ms+ delay. For this problem, we found the reason is related to vl_api_rpc_call_main_thread only put then event to master thread vl_input_queue, but does not notify and wake up master thread timely, the main thread can only process this when linux_epoll_input returns, while it will randomly sleep 10e3, if we change this value, the random ping delay will change too. We want to hear feedback from the list and hope we can find a better way to fix this. ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] 答复: 【vpp-dev】delay is error in ping with multi worker thread
I didn't investigate the code but there's an assumption I would like to share. I encountered the same problem before, but what interested me is that I found if you send multiple ping pkts as fast as you can, the problem went away. I tried different numbers and if it's a multiple of 4, VPP works just as expected. I just assumed that in order to perform “vector processing”, VPP first waits for enough( > 4) pkts to store in a vector and then, process the batch. If you send ICMP pkt one by one, VPP will only start to process these pkts after receiving at lease 4 pkts. So for the first ping, the latency is 1000ms. Best Regards, Pan zhang...@yunshan.net.cn 发件人： 张东亚 发送时间： 2017-06-22 11:09 收件人： 雷彦章 抄送： vpp-dev 主题： Re: [vpp-dev]答复: 【vpp-dev】delay is error in ping with multi worker thread Hi List, I am colleague of Leiyanzhang, let me describe share the issues and investigations found when we test the ping command under multiple worker thread case. Stage1: Issue: Ping does not work under multiple worker thread case. Version: Vanilla vpp 17.04 (vanilla vp) Stage2: Issue: Ping got > 1s delay result under multiple worker thread case, 1s is the default ping interval, if we specify 10s interval, the delay will be 10s delay. Version: Vanilla vpp 17.04 + VPP-844 patch. Our Solution: as Lei pointed out, we think it's not correct for the worker thread to directly using the ping_run_t's cli_process_id to notify the main thread because the vlib_process_signal_event is using work thread vlib_main structure, however since process is shared by main thread and worker, when cli process in main thread timeout waiting event, it can still fetch the event, but the delay will be the interval. State3: Issue: Ping delay is not related with interval, but got 10ms+ delay randomly. Version: Vanilla vpp 17.04 + VPP-844 Patch + change to using vl_api_rpc_call_main_thread to notify cli process in main thread. Our solution: After read vpp code, we think worker thread should use vl_api_rpc_call_main_thread to notify cli process in main thread, after will made the fix, the ping command works as expected most time with only random 10ms+ delay. For this problem, we found the reason is related to vl_api_rpc_call_main_thread only put then event to master thread vl_input_queue, but does not notify and wake up master thread timely, the main thread can only process this when linux_epoll_input returns, while it will randomly sleep 10e3, if we change this value, the random ping delay will change too. We want to hear feedback from the list and hope we can find a better way to fix this. ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] ACL Match in fa_node.c
Hi Andrew, I am now able to access to all the resources you shared with me, thank you! I have read the ACL plugin description file and it really helps a lot. Besides, the new high performance ACL inplementation seems interesting too :D I will be able to provide feedbacks after gaining some hand-on experiences, looking forwared to collaborating with you :p Best Regards, Pan zhang...@yunshan.net.cn From: Andrew Yourtchenko Date: 2017-05-24 02:48 To: 张攀 CC: vpp-dev Subject: Re: [vpp-dev] ACL Match in fa_node.c Hi Pan! On 5/23/17, 张攀 <zhang...@yunshan.net.cn> wrote: > Hi Andrew! > > > -- Original -- > From: "Andrew Yourtchenko"<ayour...@gmail.com>; > Date: Tue, May 23, 2017 07:56 PM > To: "张攀"<zhang...@yunshan.net.cn>; > Cc: "vpp-dev"<vpp-dev@lists.fd.io>; > Subject: Re: [vpp-dev] ACL Match in fa_node.c > > > Hi! > > On 5/23/17, 张攀 <zhang...@yunshan.net.cn> wrote: >> Hi guys, >> >> >> I looked into the source code of vpp/src/plugin/acl/fa_node.c, >> in function full_acl_match_5tuple(), it seems that every ingress packet >> is >> matching against each ACL rule stored in acl_main->acls in a for-loop >> manner. This seems not fairly effective. > > You're absolutely right on both counts. First make it work, then make > it right, then make it fast :-) I have some ideas that I wanted to > experiment with there, would you be interested to help ? ACL matching > is a fairly distinct function operation to not affect much else. > > [PAN]: I would be very pleased to help as I am also a addicted person to > high performance programming :D > > Great! :-) I will try to sketch the idea tomorrow/thursday and will add you to the draft, so we can work together. (I also will have limited connectivity the next week, so I won't be a lot in your way! :-) (or if you have some good idea on how you'd like to do it, feel free to shoot a doc/code into a gerrit draft, let's see!) > >> >> >> Besides, I notice that in vpp/src/plugin/acl/acl.c，when you call the >> function acl_hook_l2_input_classify(), you will create a >> vnet_classify_table, but I didn't see any code which adds >> classify_session >> to it, why? > > I had used classify table for storing the sessions in the pre-1704 > version of the ACL plugin. > > in 1704 as I was adding the L3, I moved to the new data path while > keeping the old one still around, and potentially switchable (not > terribly gracefully, but still). > > In the current master the classifier is used merely as a hook to get > into the packet processing within the L2 packet path - that's why you > see no sessions added. > > [PAN]: Cool. Correct me if I am wrong: ingress packet will first check > against if there is matched session existing > in fa_node.c and if not, the packet will check against ACL then to decide > whether to create a new session or drop. Exactly, that is the idea! > > > Is there any 'match then action' alike behavior existing in fa_node? And a > long standing mystery question for me: what > does "fa" stand for :p > "FA" is "feature arc" :) When adding the routed path, the feature arc support appeared a bit earlier, and I was "wow, this is so excellent and easy", and I thought I might use the same mechanism in L2 as well, so I just bluntly called everything "fa_*". But then I decided to leave for L2 data path the old mechanism of (ab)using the L2 classifier, since it allowed for a while to retain (just in case) the ability to switch between the old and the new data path. As the time goes, I suspect there might be a better way to hook in L2 forwarding path... > > >> >> >> Is there any document/idea could basically explain the relationships >> between >> acl/fa_node and vnet_classify? > > In a gerrit draft that I am working now on, which aims to provide the > multicore support, there is text file with the history and the > description of operation. If you like, I can add you to the reviewers > list so you can have a look at that - just send me your email that you > use in gerrit. > > [PAN]: Thank you Andrew, I believe this will help a lot. My email is > dazhang...@gmail.com > Excellent, I've just added you, it's 6771. As you notice, it's still a good amount of work to do (lots of clib_warning()s to ensure everything indeed does what I think it does). Looking forward to your feedback! :-) --a > > > --a > >> >> >> Any help will be much appreciated. >> >> >> Best Regards, >> >> >> Pan ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
