[vpp-dev] [VCL] Memory access error for different size of mutex with different glibc versions in VPP and VCL app
Hi All,Now, VCL app and VPP shared some data structures, such as svm_queue_t. In svm_queue_t, there are mutex and condvar variables that depends on specified glibc version. When VPP run in host and VCL app run in a docker container, glibc versions maybe different between VPP and VCL app, and then result in memory access error for different size of mutex and condvar.Has anyone noticed this?Regards,Hanlin wanghanlin wanghan...@corp.netease.com 签名由 网易邮箱大师 定制 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15839): https://lists.fd.io/g/vpp-dev/message/15839 Mute This Topic: https://lists.fd.io/mt/72485607/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [VCL] Memory access error for different size of mutex with different glibc versions in VPP and VCL app
Hi Florin,It's not only regarding compiled with the same glibc version, but running with the same glibc version also because libpthread is dynamically linked into VCL and VPP.
This is really a big limitation.Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 3/23/2020 23:31，Florin Coras wrote：
Hi Hanlin, Unfortunately, you’ll have to make sure all code has been compiled with the same glibc version. I’ve heard that glibc changed in ubuntu 20.04 but I haven’t done any testing with it yet. Note that the binary api also makes use of svm_queue_t. Regards,FlorinOn Mar 22, 2020, at 10:49 PM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi All,Now, VCL app and VPP shared some data structures, such as svm_queue_t. In svm_queue_t, there are mutex and condvar variables that depends on specified glibc version. When VPP run in host and VCL app run in a docker container, glibc versions maybe different between VPP and VCL app, and then result in memory access error for different size of mutex and condvar.Has anyone noticed this?Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15845): https://lists.fd.io/g/vpp-dev/message/15845
Mute This Topic: https://lists.fd.io/mt/72485607/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Can "use_mq_eventfd" solve epoll_wait high cpu usage problem?
hi ALL，
I found app using VCL "epoll_wait" still occupy 70% cpu with "use_mq_eventfd" configuration even if very little traffic.Then I investigate code in ldp_epoll_pwait, vls_epoll_wait is called with timeout equal to 0.Then I have two questions:1. What problems can "use_mq_eventfd" solve?2.Any other way to decrease cpu usage?Thanks!code in ldp_epoll_pwait:do { if (!ldpw->epoll_wait_vcl) { rv = vls_epoll_wait (ep_vlsh, events, maxevents, 0); if (rv > 0) { ldpw->epoll_wait_vcl = 1; goto done; } else if (rv < 0) { errno = -rv; rv = -1; goto done; } } else ldpw->epoll_wait_vcl = 0; if (libc_epfd > 0) { rv = libc_epoll_pwait (libc_epfd, events, maxevents, 0, sigmask); if (rv != 0) goto done; } } while ((timeout == -1) || (clib_time_now (&ldpw->clib_time) < max_time));
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14413): https://lists.fd.io/g/vpp-dev/message/14413
Mute This Topic: https://lists.fd.io/mt/40123765/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h =?utf-8?Q?igh_cpu_usage_problem??=
Do you mean, if just use eventfds only, then I needn't set timeout to 0 in ldp_epoll_pwait?If so, then how to process unhandled_evts_vector in vppcom_epoll_wait timely? What I'm saying is, another thread add event to unhandled_evts_vector during epoll_wait, or unhandled_evts_vector not process completely because of reaching maxevents.Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 10/31/2019 23:34，Florin Coras wrote：
Hi, use_mq_eventfd will help with vcl but as you’ve noticed it won’t help for ldp because there we need to poll both vcl and linux fds. Because mutex-condvar notifications can’t be epolled we have to constantly switch between linux and vcl epolled fds. One option going forward would be to change ldp to detect if vcl is using mutex-condvars or eventfds and in case of the latter poll linux fds and the mq’s eventfd in a linux epoll. Regards,FlorinOn Oct 31, 2019, at 5:54 AM, wanghanlin <wanghan...@corp.netease.com> wrote:hi ALL，I found app using VCL "epoll_wait" still occupy 70% cpu with "use_mq_eventfd" configuration even if very little traffic.Then I investigate code in ldp_epoll_pwait, vls_epoll_wait is called with timeout equal to 0.Then I have two questions:1. What problems can "use_mq_eventfd" solve?2.Any other way to decrease cpu usage?Thanks!code in ldp_epoll_pwait:do { if (!ldpw->epoll_wait_vcl) { rv = vls_epoll_wait (ep_vlsh, events, maxevents, 0); if (rv > 0) { ldpw->epoll_wait_vcl = 1; goto done; } else if (rv < 0) { errno = -rv; rv = -1; goto done; } } else ldpw->epoll_wait_vcl = 0; if (libc_epfd > 0) { rv = libc_epoll_pwait (libc_epfd, events, maxevents, 0, sigmask); if (rv != 0) goto done; } } while ((timeout == -1) || (clib_time_now (&ldpw->clib_time) < max_time));wanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制-=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.View/Reply Online (#14413): https://lists.fd.io/g/vpp-dev/message/14413Mute This Topic: https://lists.fd.io/mt/40123765/675152Group Owner: vpp-dev+ow...@lists.fd.ioUnsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]-=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14448): https://lists.fd.io/g/vpp-dev/message/14448
Mute This Topic: https://lists.fd.io/mt/40351193/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h =?utf-8?Q?_igh_cpu_usage_problem??=
OK，I got it. Thanks a lot.By the way, can VCL adapt to envoy or any progress about this？
Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 11/1/2019 12:07，Florin Coras wrote：
Hi Hanlin, If a worker’s mq uses eventfds for notifications, we could nest it in libc_epfd, i.e.,the epoll fd we create for the linux fds. So, if an app's worker calls epoll_wait, in ldp we can epoll_wait on libc_epfd and if we get an event on the mq’s eventfd, we can call vls_epoll_wait with a 0 timeout to drain the events from vcl. Having said that, keep in mind that we typically recommend that people use vcl because ldp, through vls, enforces a rather strict locking policy. That is needed in order to avoid invalidating vcl’s assumption that sessions are owned by only one vcl worker. Moreover, we’ve tested ldp only against a limited set of applications. Regards, FlorinOn Oct 31, 2019, at 7:58 PM, wanghanlin <wanghan...@corp.netease.com> wrote:Do you mean, if just use eventfds only, then I needn't set timeout to 0 in ldp_epoll_pwait?If so, then how to process unhandled_evts_vector in vppcom_epoll_wait timely? What I'm saying is, another thread add event to unhandled_evts_vector during epoll_wait, or unhandled_evts_vector not process completely because of reaching maxevents.Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 10/31/2019 23:34，Florin Coras wrote： Hi, use_mq_eventfd will help with vcl but as you’ve noticed it won’t help for ldp because there we need to poll both vcl and linux fds. Because mutex-condvar notifications can’t be epolled we have to constantly switch between linux and vcl epolled fds. One option going forward would be to change ldp to detect if vcl is using mutex-condvars or eventfds and in case of the latter poll linux fds and the mq’s eventfd in a linux epoll. Regards,FlorinOn Oct 31, 2019, at 5:54 AM, wanghanlin <wanghan...@corp.netease.com> wrote:hi ALL，I found app using VCL "epoll_wait" still occupy 70% cpu with "use_mq_eventfd" configuration even if very little traffic.Then I investigate code in ldp_epoll_pwait, vls_epoll_wait is called with timeout equal to 0.Then I have two questions:1. What problems can "use_mq_eventfd" solve?2.Any other way to decrease cpu usage?Thanks!code in ldp_epoll_pwait:do { if (!ldpw->epoll_wait_vcl) { rv = vls_epoll_wait (ep_vlsh, events, maxevents, 0); if (rv > 0) { ldpw->epoll_wait_vcl = 1; goto done; } else if (rv < 0) { errno = -rv; rv = -1; goto done; } } else ldpw->epoll_wait_vcl = 0; if (libc_epfd > 0) { rv = libc_epoll_pwait (libc_epfd, events, maxevents, 0, sigmask); if (rv != 0) goto done; } } while ((timeout == -1) || (clib_time_now (&ldpw->clib_time) < max_time));wanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制-=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.View/Reply Online (#14413): https://lists.fd.io/g/vpp-dev/message/14413Mute This Topic: https://lists.fd.io/mt/40123765/675152Group Owner: vpp-dev+ow...@lists.fd.ioUnsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.View/Reply Online (#14448): https://lists.fd.io/g/vpp-dev/message/14448Mute This Topic: https://lists.fd.io/mt/40351193/675152Group Owner: vpp-dev+ow...@lists.fd.ioUnsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]-=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14451): https://lists.fd.io/g/vpp-dev/message/14451
Mute This Topic: https://lists.fd.io/mt/40377434/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h igh cpu usage problem?
Hi stephenIt's great. Is there any performance data comparing with the kernel path now?
Regards,Hanlin
发自网易邮箱大师
On 11/2/2019 06:42，Stephen Belair (sbelair) via Lists.Fd.Io wrote：
Hanlin,
I’ve been working on Envoy/Vcl integration for a while now and Ping Yu has joined me in this effort. Vcl works fine with a large, multi-threaded application like Envoy, although the integration has had its challenges. We are getting very
close to trying to upstream the code.
-stephen
From: Florin Coras
Date: Thursday, October 31, 2019 at 10:26 PM
To: wanghanlin , "Stephen Belair (sbelair)" , "Yu, Ping"
Cc: "vpp-dev@lists.fd.io"
Subject: Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h igh cpu usage problem?
Hi Hanlin,
Stephen and Ping have made a lot of progress with Envoy and VCL, but I’ll let them comment on that.
Regards,
Florin
On Oct 31, 2019, at 9:44 PM, wanghanlin <wanghan...@corp.netease.com> wrote:
OK，I got it. Thanks a lot.
By the way, can VCL adapt to envoy or any progress about this？
Regards,
Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由 网易邮箱大师 定制
On 11/1/2019 12:07，Florin
Coras wrote：
Hi Hanlin,
If a worker’s mq uses eventfds for notifications, we could nest it in libc_epfd, i.e.,the epoll fd we create for the linux fds. So, if an app's worker calls epoll_wait,
in ldp we can epoll_wait on libc_epfd and if we get an event on the mq’s eventfd, we can call vls_epoll_wait with a 0 timeout to drain the events from vcl.
Having said that, keep in mind that we typically recommend that people use vcl because ldp, through vls, enforces a rather strict locking policy. That is needed in
order to avoid invalidating vcl’s assumption that sessions are owned by only one vcl worker. Moreover, we’ve tested ldp only against a limited set of applications.
Regards,
Florin
On Oct 31, 2019, at 7:58 PM, wanghanlin <wanghan...@corp.netease.com> wrote:
Do you mean, if just use eventfds only, then I needn't set timeout to 0 in ldp_epoll_pwait?
If so, then how to process unhandled_evts_vector in vppcom_epoll_wait timely? What I'm saying is, another thread add event to unhandled_evts_vector during epoll_wait,
or unhandled_evts_vector not process completely because of reaching maxevents.
Regards,
Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由 网易邮箱大师 定制
On 10/31/2019 23:34，Florin
Coras wrote：
Hi,
use_mq_eventfd will help with vcl but as you’ve noticed it won’t help for ldp because there we need to poll both vcl and linux fds. Because mutex-condvar notifications
can’t be epolled we have to constantly switch between linux and vcl epolled fds. One option going forward would be to change ldp to detect if vcl is using mutex-condvars or eventfds and in case of the latter poll linux fds and the mq’s eventfd in a linux epoll.
Regards,
Florin
On Oct 31, 2019, at 5:54 AM, wanghanlin <wanghan...@corp.netease.com> wrote:
hi ALL，
I found app using VCL "epoll_wait" still occupy 70% cpu with "use_mq_eventfd" configuration even if very little traffic.
Then I investigate code in ldp_epoll_pwait, vls_epoll_wait is called with timeout equal to 0.
Then I have two questions:
1. What problems can "use_mq_eventfd" solve?
2.Any other way to decrease cpu usage?
Thanks!
code in ldp_epoll_pwait:
do
{
if (!ldpw->epoll_wait_vcl)
{
rv = vls_epoll_wait (ep_vlsh, events, maxevents,
0);
if (rv > 0)
{
ldpw->epoll_wait_vcl = 1;
goto done;
}
else if (rv < 0)
{
errno = -rv;
rv = -1;
goto done;
}
}
else
ldpw->epoll_wait_vcl = 0;
if (libc_epfd > 0)
{
rv = libc_epoll_pwait (libc_epfd, events,
maxevents, 0, sigmask);
if (rv != 0)
goto done;
}
}
while ((timeout == -1) || (clib_time_now (&ldpw->clib_time) < max_time));
wanghanlin
wanghan...@corp.netease.com
签名由 网易邮箱大师 定制
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14413): https://lists.fd.io/g/vpp-dev/message/14413
Mute This Topic: https://lists.fd.io/mt/40123765/675152
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]
-=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14448): https://lists.fd.io/g/vpp-dev/message/14448
Mute This Topic: https://lists.fd.io/mt/40351193/675152
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]
-=-=
Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h =?utf-8?Q?_igh_cpu_usage_problem??=
OK，Thanks.
Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 11/5/2019 00:12，Stephen Belair (sbelair) wrote：
Hi Hanlin,
We are working on getting perf numbers and should have some comparisons in a few weeks.
Thanks,
-stephen
From: wanghanlin
Date: Friday, November 1, 2019 at 6:50 PM
To: "Stephen Belair (sbelair)"
Cc: Florin Coras , "Yu, Ping" , "vpp-dev@lists.fd.io"
Subject: Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h igh cpu usage problem?
Hi stephen
It's great. Is there any performance data comparing with the kernel path now?
Regards,
Hanlin
发自网易邮箱大师
On
11/2/2019 06:42，Stephen Belair (sbelair) via Lists.Fd.Io wrote：
Hanlin,
I’ve been working on Envoy/Vcl integration for a while now and Ping Yu has joined me in this effort. Vcl works fine with a large, multi-threaded application like Envoy, although the integration has had its challenges. We are getting very
close to trying to upstream the code.
-stephen
From: Florin Coras
Date: Thursday, October 31, 2019 at 10:26 PM
To: wanghanlin , "Stephen Belair (sbelair)" , "Yu, Ping"
Cc: "vpp-dev@lists.fd.io"
Subject: Re: [vpp-dev] Can "use mq eventfd" solve epoll wait h igh cpu usage problem?
Hi Hanlin,
Stephen and Ping have made a lot of progress with Envoy and VCL, but I’ll let them comment on that.
Regards,
Florin
On Oct 31, 2019, at 9:44 PM, wanghanlin <wanghan...@corp.netease.com> wrote:
OK，I got it. Thanks a lot.
By the way, can VCL adapt to envoy or any progress about this？
Regards,
Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由 网易邮箱大师 定制
On 11/1/2019 12:07，Florin
Coras wrote：
Hi Hanlin,
If a worker’s mq uses eventfds for notifications, we could nest it in libc_epfd, i.e.,the epoll fd we create for the linux fds. So, if an app's worker calls epoll_wait,
in ldp we can epoll_wait on libc_epfd and if we get an event on the mq’s eventfd, we can call vls_epoll_wait with a 0 timeout to drain the events from vcl.
Having said that, keep in mind that we typically recommend that people use vcl because ldp, through vls, enforces a rather strict locking policy. That is needed in
order to avoid invalidating vcl’s assumption that sessions are owned by only one vcl worker. Moreover, we’ve tested ldp only against a limited set of applications.
Regards,
Florin
On Oct 31, 2019, at 7:58 PM, wanghanlin <wanghan...@corp.netease.com> wrote:
Do you mean, if just use eventfds only, then I needn't set timeout to 0 in ldp_epoll_pwait?
If so, then how to process unhandled_evts_vector in vppcom_epoll_wait timely? What I'm saying is, another thread add event to unhandled_evts_vector during epoll_wait,
or unhandled_evts_vector not process completely because of reaching maxevents.
Regards,
Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由 网易邮箱大师 定制
On 10/31/2019 23:34，Florin
Coras wrote：
Hi,
use_mq_eventfd will help with vcl but as you’ve noticed it won’t help for ldp because there we need to poll both vcl and linux fds. Because mutex-condvar notifications
can’t be epolled we have to constantly switch between linux and vcl epolled fds. One option going forward would be to change ldp to detect if vcl is using mutex-condvars or eventfds and in case of the latter poll linux fds and the mq’s eventfd in a linux epoll.
Regards,
Florin
On Oct 31, 2019, at 5:54 AM, wanghanlin <wanghan...@corp.netease.com> wrote:
hi ALL，
I found app using VCL "epoll_wait" still occupy 70% cpu with "use_mq_eventfd" configuration even if very little traffic.
Then I investigate code in ldp_epoll_pwait, vls_epoll_wait is called with timeout equal to 0.
Then I have two questions:
1. What problems can "use_mq_eventfd" solve?
2.Any other way to decrease cpu usage?
Thanks!
code in ldp_epoll_pwait:
do
{
if (!ldpw->epoll_wait_vcl)
{
rv = vls_epoll_wait (ep_vlsh, events, maxevents,
0);
if (rv > 0)
{
ldpw->epoll_w
[vpp-dev] [VCL] hoststack app crash with invalid memfd segment address
hi ALL,I accidentally got following crash stack when I used VCL with hoststack and memfd. But corresponding invalid rx_fifo address (0x2f42e2480) is valid in VPP process and also can be found in /proc/map. That is, shared memfd segment memory is not consistent between hoststack app and VPP.Generally, VPP allocate/dealloc the memfd segment and then notify hoststack app to attach/detach. But If just after VPP dealloc memfd segment and notify hoststack app, and then VPP allocate same memfd segment at once because of session connected, and then what happened now? Because hoststack app process dealloc message and connected message with diffrent threads, maybe rx_thread_fn just detach the memfd segment and not attach the same memfd segment, then unfortunately worker thread get the connected message. These are just my guess, maybe I misunderstand.(gdb) bt#0 0x7f7cde21ffbf in raise () from /lib/x86_64-linux-gnu/libpthread.so.0#1 0x01190a64 in Envoy::SignalAction::sigHandler (sig=11, info=, context=) at source/common/signal/signal_action.cc:73#2 #3 0x7f7cddc2e85e in vcl_session_connected_handler (wrk=0x7f7ccd4bad00, mp=0x224052f4a) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:471#4 0x7f7cddc37fec in vcl_epoll_wait_handle_mq_event (wrk=0x7f7ccd4bad00, e=0x224052f48, events=0x395000c, num_ev=0x7f7cca49e5e8) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2658#5 0x7f7cddc3860d in vcl_epoll_wait_handle_mq (wrk=0x7f7ccd4bad00, mq=0x224042480, events=0x395000c, maxevents=63, wait_for_time=0, num_ev=0x7f7cca49e5e8) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2762#6 0x7f7cddc38c74 in vppcom_epoll_wait_eventfd (wrk=0x7f7ccd4bad00, events=0x395000c, maxevents=63, n_evts=0, wait_for_time=0) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2823#7 0x7f7cddc393a0 in vppcom_epoll_wait (vep_handle=33554435, events=0x395000c, maxevents=63, wait_for_time=0) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2880#8 0x7f7cddc5d659 in vls_epoll_wait (ep_vlsh=3, events=0x395000c, maxevents=63, wait_for_time=0) at /home/wanghanlin/vpp-new/src/vcl/vcl_locked.c:895#9 0x7f7cdeb4c252 in ldp_epoll_pwait (epfd=67, events=0x395, maxevents=64, timeout=32, sigmask=0x0) at /home/wanghanlin/vpp-new/src/vcl/ldp.c:2334#10 0x7f7cdeb4c334 in epoll_wait (epfd=67, events=0x395, maxevents=64, timeout=32) at /home/wanghanlin/vpp-new/src/vcl/ldp.c:2389#11 0x00fc9458 in epoll_dispatch ()#12 0x00fc363c in event_base_loop ()#13 0x00c09b1c in Envoy::Server::WorkerImpl::threadRoutine (this=0x357d8c0, guard_dog=...) at source/server/worker_impl.cc:104#14 0x01193485 in std::function::operator()() const (this=0x7f7ccd4b8544) at /usr/lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/std_function.h:706#15 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function)::$_0::operator()(void*) const (this=, arg=0x2f42e2480) at source/common/common/posix/thread_impl.cc:33#16 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function)::$_0::__invoke(void*) (arg=0x2f42e2480) at source/common/common/posix/thread_impl.cc:32#17 0x7f7cde2164a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0#18 0x7f7cddf58d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6(gdb) f 3#3 0x7f7cddc2e85e in vcl_session_connected_handler (wrk=0x7f7ccd4bad00, mp=0x224052f4a) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:471471 rx_fifo->client_session_index = session_index;(gdb) p rx_fifo$1 = (svm_fifo_t *) 0x2f42e2480(gdb) p *rx_fifoCannot access memory at address 0x2f42e2480(gdb)Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14604): https://lists.fd.io/g/vpp-dev/message/14604
Mute This Topic: https://lists.fd.io/mt/59126583/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [VCL] hoststack app crash with invalid memfd segment address
Hi Florin,VPP vsersion is v19.08.I'll apply this patch and check it. Thanks a lot!
Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 11/16/2019 00:50，Florin Coras wrote：
Hi Hanlin,Just to make sure, are you running master or some older VPP?Regarding the issue you could be hitting lower, here’s [1] a patch that I have not yet pushed for merging because it leads to api changes for applications that directly use the session layer application interface instead of vcl. I haven’t tested it extensively, but the goal with it is to signal segment allocation/deallocation over the mq instead of the binary api.Finally, I’ve never tested LDP with Envoy, so not sure if that works properly. There’s ongoing work to integrate Envoy with VCL, so you may want to get in touch with the authors. Regards,Florin[1] https://gerrit.fd.io/r/c/vpp/+/21497On Nov 15, 2019, at 2:26 AM, wanghanlin <wanghan...@corp.netease.com> wrote:hi ALL,I accidentally got following crash stack when I used VCL with hoststack and memfd. But corresponding invalid rx_fifo address (0x2f42e2480) is valid in VPP process and also can be found in /proc/map. That is, shared memfd segment memory is not consistent between hoststack app and VPP.Generally, VPP allocate/dealloc the memfd segment and then notify hoststack app to attach/detach. But If just after VPP dealloc memfd segment and notify hoststack app, and then VPP allocate same memfd segment at once because of session connected, and then what happened now? Because hoststack app process dealloc message and connected message with diffrent threads, maybe rx_thread_fn just detach the memfd segment and not attach the same memfd segment, then unfortunately worker thread get the connected message. These are just my guess, maybe I misunderstand.(gdb) bt#0 0x7f7cde21ffbf in raise () from /lib/x86_64-linux-gnu/libpthread.so.0#1 0x01190a64 in Envoy::SignalAction::sigHandler (sig=11, info=, context=) at source/common/signal/signal_action.cc:73#2 #3 0x7f7cddc2e85e in vcl_session_connected_handler (wrk=0x7f7ccd4bad00, mp=0x224052f4a) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:471#4 0x7f7cddc37fec in vcl_epoll_wait_handle_mq_event (wrk=0x7f7ccd4bad00, e=0x224052f48, events=0x395000c, num_ev=0x7f7cca49e5e8) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2658#5 0x7f7cddc3860d in vcl_epoll_wait_handle_mq (wrk=0x7f7ccd4bad00, mq=0x224042480, events=0x395000c, maxevents=63, wait_for_time=0, num_ev=0x7f7cca49e5e8) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2762#6 0x7f7cddc38c74 in vppcom_epoll_wait_eventfd (wrk=0x7f7ccd4bad00, events=0x395000c, maxevents=63, n_evts=0, wait_for_time=0) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2823#7 0x7f7cddc393a0 in vppcom_epoll_wait (vep_handle=33554435, events=0x395000c, maxevents=63, wait_for_time=0) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:2880#8 0x7f7cddc5d659 in vls_epoll_wait (ep_vlsh=3, events=0x395000c, maxevents=63, wait_for_time=0) at /home/wanghanlin/vpp-new/src/vcl/vcl_locked.c:895#9 0x7f7cdeb4c252 in ldp_epoll_pwait (epfd=67, events=0x395, maxevents=64, timeout=32, sigmask=0x0) at /home/wanghanlin/vpp-new/src/vcl/ldp.c:2334#10 0x7f7cdeb4c334 in epoll_wait (epfd=67, events=0x395, maxevents=64, timeout=32) at /home/wanghanlin/vpp-new/src/vcl/ldp.c:2389#11 0x00fc9458 in epoll_dispatch ()#12 0x00fc363c in event_base_loop ()#13 0x00c09b1c in Envoy::Server::WorkerImpl::threadRoutine (this=0x357d8c0, guard_dog=...) at source/server/worker_impl.cc:104#14 0x01193485 in std::function::operator()() const (this=0x7f7ccd4b8544) at /usr/lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/std_function.h:706#15 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function)::$_0::operator()(void*) const (this=, arg=0x2f42e2480) at source/common/common/posix/thread_impl.cc:33#16 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::function)::$_0::__invoke(void*) (arg=0x2f42e2480) at source/common/common/posix/thread_impl.cc:32#17 0x7f7cde2164a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0#18 0x7f7cddf58d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6(gdb) f 3#3 0x7f7cddc2e85e in vcl_session_connected_handler (wrk=0x7f7cc
Re: [vpp-dev] [VCL] hoststack app crash with invalid memfd segment address
Hi Florin,
I have applied the patch, and found some problems in my case. I have not right to post it in gerrit, so I post here.
1)evt->event_type should be set with SESSION_CTRL_EVT_APP_DEL_SEGMENT rather than SESSION_CTRL_EVT_APP_ADD_SEGMENT. File: src/vnet/session/session_api.c, Line: 561, Function:mq_send_del_segment_cb2)session_send_fds may been called in the end of function mq_send_add_segment_cb, otherwise lock of app_mq can't been free here.File: src/vnet/session/session_api.c, Line: 519, Function:mq_send_add_segment_cb
3) When vcl_segment_attach called in each worker thread, then ssvm_slave_init_memfd can been called in each worker thread and then ssvm_slave_init_memfd map address sequentially through map segment once in advance. It's OK in only one thread, but maybe wrong in multiple worker threads. Suppose following scene: VPP allocate segment at address A1 and notify worker thread B1 to expect B1 also map segment at address A1, and simultaneously VPP allocate segment at address A2 and notify worker thread B2 to expect B2 map segment at address A2. If B2 first process notify message, then ssvm_slave_init_memfd may map segment at address A1. Maybe VPP can add segment map address in notify message, and then worker thread just map segment at this address. Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 11/19/2019 09:50，wanghanlin wrote：
Hi Florin,VPP vsersion is v19.08.I'll apply this patch and check it. Thanks a lot!
Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 11/16/2019 00:50，Florin Coras wrote：
Hi Hanlin,Just to make sure, are you running master or some older VPP?Regarding the issue you could be hitting lower, here’s [1] a patch that I have not yet pushed for merging because it leads to api changes for applications that directly use the session layer application interface instead of vcl. I haven’t tested it extensively, but the goal with it is to signal segment allocation/deallocation over the mq instead of the binary api.Finally, I’ve never tested LDP with Envoy, so not sure if that works properly. There’s ongoing work to integrate Envoy with VCL, so you may want to get in touch with the authors. Regards,Florin[1] https://gerrit.fd.io/r/c/vpp/+/21497On Nov 15, 2019, at 2:26 AM, wanghanlin <wanghan...@corp.netease.com> wrote:hi ALL,I accidentally got following crash stack when I used VCL with hoststack and memfd. But corresponding invalid rx_fifo address (0x2f42e2480) is valid in VPP process and also can be found in /proc/map. That is, shared memfd segment memory is not consistent between hoststack app and VPP.Generally, VPP allocate/dealloc the memfd segment and then notify hoststack app to attach/detach. But If just after VPP dealloc memfd segment and notify hoststack app, and then VPP allocate same memfd segment at once because of session connected, and then what happened now? Because hoststack app process dealloc message and connected message with diffrent threads, maybe rx_thread_fn just detach the memfd segment and not attach the same memfd segment, then unfortunately worker thread get the connected message. These are just my guess, maybe I misunderstand.(gdb) bt#0 0x7f7cde21ffbf in raise () from /lib/x86_64-linux-gnu/libpthread.so.0#1 0x01190a64 in Envoy::SignalAction::sigHandler (sig=11, info=, context=) at source/common/signal/signal_action.cc:73#2 #3 0x7f7cddc2e85e in vcl_session_connected_handler (wrk=0x7f7ccd4bad00, mp=0x224052f4a) at /home/wanghanlin/vpp-new/src/vcl/vppcom.c:471#4 0x7f7cddc37fec in vcl_epoll_wait_handle_mq_event (wrk=0x7f7
Re: [vpp-dev] [VCL] hoststack app crash with invalid memfd segment address
Hi Florin,
Regarding 3), I think main problem maybe in function vl_socket_client_recv_fd_msg called by vcl_session_app_add_segment_handler. Mutiple worker threads share the same scm->client_socket.fd, so B2 may receive the segment memfd belong to A1. Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 11/22/2019 01:44，Florin Coras wrote：
Hi Hanlin, As Jon pointed out, you may want to register with gerrit. You comments with respect to points 1) and 2) are spot on. I’ve updated the patch to fix them. Regarding 3), if I understood your scenario correctly, it should not happen. The ssvm infra forces applications to map segments at fixed addresses. That is, for the scenario you’re describing lower, if B2 is processed first, ssvm_slave_init_memfd will map the segment at A2. Note how we first map the segment to read the shared header (sh) and then use sh->ssvm_va (which should be A2) to remap the segment at a fixed virtual address (va). Regards,FlorinOn Nov 21, 2019, at 2:49 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,I have applied the patch, and found some problems in my case. I have not right to post it in gerrit, so I post here.1)evt->event_type should be set with SESSION_CTRL_EVT_APP_DEL_SEGMENT rather than SESSION_CTRL_EVT_APP_ADD_SEGMENT. File: src/vnet/session/session_api.c, Line: 561, Function:mq_send_del_segment_cb2)session_send_fds may been called in the end of function mq_send_add_segment_cb, otherwise lock of app_mq can't been free here.File: src/vnet/session/session_api.c, Line: 519, Function:mq_send_add_segment_cb 3) When vcl_segment_attach called in each worker thread, then ssvm_slave_init_memfd can been called in each worker thread and then ssvm_slave_init_memfd map address sequentially through map segment once in advance. It's OK in only one thread, but maybe wrong in multiple worker threads. Suppose following scene: VPP allocate segment at address A1 and notify worker thread B1 to expect B1 also map segment at address A1, and simultaneously VPP allocate segment at address A2 and notify worker thread B2 to expect B2 map segment at address A2. If B2 first process notify message, then ssvm_slave_init_memfd may map segment at address A1. Maybe VPP can add segment map address in notify message, and then worker thread just map segment at this address. Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/19/2019 09:50，wanghanlin wrote： Hi Florin,VPP vsersion is v19.08.I'll apply this patch and check it. Thanks a lot!Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/16/2019 00:50，Florin Coras wrote： Hi Hanlin,Just to make sure, are you running master or some older VPP?Regarding the issue you could be hitting lower, here’s [1] a patch that I have not yet pushed for merging because it leads to api changes for applications that directly use the session layer application interface instead of vcl. I haven’t tested it extensively, but the goal with it is to signal segment allocation/deallocation over the mq instead of the binary api.Finally, I’ve never tested LDP with Envoy, so not sure if that works properly. There’s ongoing work to integrate Envoy with VCL, so you may want to get in touch with the authors. Regards,Florin[1] https://gerrit.fd.io/r/c/vpp/+/21497On Nov 15, 2019, at 2:26 AM, wanghanlin <wanghan...@corp.netease.com> wrote:hi ALL,I accidentally got following crash stack when I used VCL with hoststack and memfd. But corresponding invalid rx_fifo address (0x2f42e2480) is valid in VPP process and also can be found in /proc/map. That is, shared memfd segment memory is not consistent between hoststack app and VPP.Generally, VPP allocate/dealloc the memfd segment and then notify hoststack app to attach/detach. But If just after VPP dealloc memfd segment and notify hoststack app, and then VPP allocate same memfd segment at once because of session connected, and then what happened now? Because hoststack app process dealloc message and connected message with diffrent threads, maybe rx_thread_fn just detach the memfd segment and not attach the same memfd segment, then unfortunately worker thread get the connected message. These are just my guess, maybe I misunderstand.(gdb) bt#0 0x7f7cde21ffb
[vpp-dev] How to configure network between different namespaces using hoststack
Hi All， We have two APPs in different namespaces, such as APP1 and APP2, and use hoststack based on same VPP. APP1 listen on 192.168.1.2:8080，and APP2 connect APP2 through 192.168.1.3. Then, we can add two interfaces on VPP, but how to configure ip address? Suppose netmask is 24, VPP can not support two interfaces with same subnet. Regards,Hanlin wanghanlin wanghan...@corp.netease.com 签名由 网易邮箱大师 定制 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14725): https://lists.fd.io/g/vpp-dev/message/14725 Mute This Topic: https://lists.fd.io/mt/64106592/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] How to configure network between different namespaces using hoststack
Hi Florin, Thanks for your reply. I just consider a very simple use case. Some apps in different containers communicate through VPP, just in a L2 bridge domain. Without hoststack, we may add some host-interfaces in one bridge domain, and assign IP address of veth interface in containers. In addition, a physical nic also added in same bridge domain to communicate with other hosts. But with hoststack, things seem complicated because we have to assign IP address inside VPP. I hope apps can communicate with each other and with external hosts in the same vrf and source ip is enforced and not changed during communication. If not, can multiple vrfs achieve this? Thanks, Hanlin -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14737): https://lists.fd.io/g/vpp-dev/message/14737 Mute This Topic: https://lists.fd.io/mt/64106592/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] How to configure network between different namespaces using hoststack
Hi Florin，Thanks for your patient reply. Still I have some doubt inline. wanghanlin wanghan...@corp.netease.com 签名由 网易邮箱大师 定制 On 11/30/2019 02:47，Florin Coras wrote： Hi Hanlin, Inline. On Nov 29, 2019, at 7:12 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin, Thanks for your reply. I just consider a very simple use case. Some apps in different containers communicate through VPP, just in a L2 bridge domain. Without hoststack, we may add some host-interfaces in one bridge domain, and assign IP address of veth interface in containers. In addition, a physical nic also added in same bridge domain to communicate with other hosts. But with hoststack, things seem complicated because we have to assign IP address inside VPP. FC: Yes, with host stack transport protocols are terminated in vpp, therefore the interfaces must have IPs. Do you need network access to the container’s linux stack for other applications, i.e., do you need IPs in the container as well? Also, can’t you give the interfaces /32 IPs?Hanlin：I need not access to contaner's linux stack now, I think I can create another host-interface with another IP if needed. Also, if I give the interfaces /32 IPs, then how to communicate with each other and external hosts? As an alternative, I assign multiple /24 IPs to one interface, then two applications can communicate with each other and external hosts, but can only get 0.0.0.0/0 source address at accept time when communicating with each other. Maybe I should bind to a IP before connect if I want to get this specified IP？ I hope apps can communicate with each other and with external hosts in the same vrf and source ip is enforced and not changed during communication. If not, can multiple vrfs achieve this?FC: If applications are attached to the same app namespace, then you could leverage cut-through connections if you enable local scope connections at attachment time (see slides 17 and 18 here [1]). Cut-through sessions are “connected” at session layer, so they don’t pass through the IP fib.Hanlin：Can local scope and global scope enable simultaneously? ie, some connections use local scope and others use global scope simultaneously.Otherwise, connectivity between the apps is established via intra-vrf or inter-vrf routing. Intra-vrf you don’t need to configure anything more, inter-vrf you need to add additional routes. For external hosts, you need routes to them in the vrfs. Hanlin：Inter-vrf leaking seems to not work when multiple vrf have same subnet IPs. In test/test_vcl.py, two vrf table have different subnet IPs.What we call “local” IPs for a connection are assigned at connect/accept time and they do not change. When connecting, we use the first IP of an interface that has a route to the destination and on accept, we use the dst IP on the SYN packet. Regards,Florin[1] https://wiki.fd.io/images/9/9c/Vpp-hoststack-kc-eu19.pdf Thanks, Hanlin -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.View/Reply Online (#14737): https://lists.fd.io/g/vpp-dev/message/14737Mute This Topic: https://lists.fd.io/mt/64106592/675152Group Owner: vpp-dev+ow...@lists.fd.ioUnsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]-=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14781): https://lists.fd.io/g/vpp-dev/message/14781 Mute This Topic: https://lists.fd.io/mt/64106592/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] How to configure network between different namespaces using hoststack
Hi Florin,
Okay, regarding first question, the following is the detailed use case:I have one 82599 nic in my Linux host. Then I allocate two VF interfaces through SRIOV, one VF place into a Linux namespace N1 and assign IP address 192.168.1.2/24, another VF place into VPP. I have three applications (just called APP1, APP2, APP3) communicating with each other, and each application must get the source IP address (not 0.0.0.0) after accept for a connect request.APP1 run in Linux namespce N1 and use IP address 192.168.1.2/24. APP2 run in Linux namespace N2 and use IP address 192.168.1.3/24. APP3 run in Linux namespace N3 and use IP address 192.168.1.4/24. And finally, APP2 and APP3 need to run based LDP.Let's summarize:APP1, N1, 192.168.1.2/24, outside VPPAPP2, N2, 192.168.1.3/24, inside VPPAPP3, N3, 192.168.1.4/24, inside VPPThen, my question is how to configure 192.168.1.3/24 and 192.168.1.4/24 in VPP?Thanks & Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 12/6/2019 03:56，Florin Coras wrote：
Hi Hanlin, Inline.On Dec 4, 2019, at 1:59 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin，Thanks for your patient reply. Still I have some doubt inline.wanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/30/2019 02:47，Florin Coras wrote： Hi Hanlin, Inline. On Nov 29, 2019, at 7:12 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,Thanks for your reply.I just consider a very simple use case. Some apps in different containers communicate through VPP, just in a L2 bridge domain. Without hoststack, we may add some host-interfaces in one bridge domain, and assign IP address of veth interface in containers. In addition, a physical nic also added in same bridge domain to communicate with other hosts.But with hoststack, things seem complicated because we have to assign IP address inside VPP. FC: Yes, with host stack transport protocols are terminated in vpp, therefore the interfaces must have IPs. Do you need network access to the container’s linux stack for other applications, i.e., do you need IPs in the container as well? Also, can’t you give the interfaces /32 IPs?Hanlin：I need not access to contaner's linux stack now, I think I can create another host-interface with another IP if needed. Also, if I give the interfaces /32 IPs, then how to communicate with each other and external hosts? FC: I’m not sure I understand the question. I’m inclined to say vpp routing and/or cut-through sessions, but it feels like I’m missing some point you were trying to make. As an alternative, I assign multiple /24 IPs to one interface, then two applications can communicate with each other and external hosts, but can only get 0.0.0.0/0 source address at accept time when communicating with each other. Maybe I should bind to a IP before connect if I want to get this specified IP？ FC: If you use /24 for the interface then, if you want a unique local IPs for each app, you should use an explicit source ip in the connect (see vppcom_session_attr and VPPCOM_ATTR_SET_LCL_ADDR).I hope apps can communicate with each other and with external hosts in the same vrf and source ip is enforced and not changed during communication. If not, can multiple vrfs achieve this?FC: If applications are attached to the same app namespace, then you could leverage cut-through connections if you enable local scope connections at attachment time (see slides 17 and 18 here [1]). Cut-through sessions are “connected” at session layer, so they don’t pass through the IP fib.Hanlin：Can local scope and global scope enable simultaneously? ie, some connections use local scope and others use global scope simultaneously.FC: Yes, you can. Otherwise, connectivity between the apps is established via intra-vrf or inter-vrf routing. Intra-vrf you don’t need to configure anything more, inter-vrf you need to add additional routes. For external hosts, you need routes to them in the vrfs. Hanlin：Inter-vrf leaking seems to not work when multiple vrf have same subnet IPs. In test/test_vcl.py, two vrf table have different subnet IPs.FC: Yes, that’s true. Let’s clarify your first question and see which of the two options (multiple /32 interfaces or a /24 one) works. Regards, FlorinWhat we call “local” IPs for a connection are assigned at connect/accept t
Re: [vpp-dev] How to configure network between different namespaces using hoststack
Hi Florin, Thanks for your suggestion.Follow your suggested configuration, three applications can communicate each other. But there are two minor problems:1. I used option 2 to configure APP2 and APP3, but can only get the source IP address 0.0.0.0 (not 192.168.1.129 or 192.168.1.130) after accept for a connect request. 2. There are two different networks between APP1 and APP2/APP3 in your configuration. But if they are all in the same network, then how to configure? Following are vpp, vcl and fib configurations:Thanks & Regards,Hanlin wanghanlin wanghan...@corp.netease.com 签名由 网易邮箱大师 定制 On 12/7/2019 07:11，Florin Coras wrote： Hi Hanlin, Inline. On Dec 5, 2019, at 7:00 PM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,Okay, regarding first question, the following is the detailed use case:I have one 82599 nic in my Linux host. Then I allocate two VF interfaces through SRIOV, one VF place into a Linux namespace N1 and assign IP address 192.168.1.2/24, another VF place into VPP. I have three applications (just called APP1, APP2, APP3) communicating with each other, and each application must get the source IP address (not 0.0.0.0) after accept for a connect request.APP1 run in Linux namespce N1 and use IP address 192.168.1.2/24. APP2 run in Linux namespace N2 and use IP address 192.168.1.3/24. APP3 run in Linux namespace N3 and use IP address 192.168.1.4/24. And finally, APP2 and APP3 need to run based LDP.Let's summarize:APP1, N1, 192.168.1.2/24, outside VPPAPP2, N2, 192.168.1.3/24, inside VPPAPP3, N3, 192.168.1.4/24, inside VPPFC: I assume N2 and N3 are mapped to app namespaces from VPP perspective. Additionally, those two prefixes, i.e., 192.168.1.3/24 and 192.168.1.4/24, do not need to be configured on interfaces part of N2 and N3 respectively. Then, from vpp perspective, APP2 and APP3 are “locally attached” and APP1 is “remote”. So, from my perspective, they’re at least two different networks. APP2 and APP3 could be the same or different networks.For instance, you could assign 192.168.1.2/25 to N1 and then leave 192.168.1.128/25 to vpp for N2 and N3. Within vpp you have two options:- add two interfaces, say intN2 and intN3 with IPs 192.168.1.129/32 and 192.168.1.130/32 and associate N2 and N3 app namespaces to those interfaces (not the fibs). Whenever initiating connections, APP2 and APP3 will pick up the ips of the interfaces associated to their respective app namespaces. - add one interface intN with IP 192.168.1.129/25 and associate both namespaces to it. If you need APP1 to use 192.168.1.129 and APP2 192.168.1.130, then you’ll need your apps to call bind before connecting (haven’t tested this but I think it should work). The above assumes APP2 and APP3 map to different app namespaces. If you want to use the same app namespace, to be able to use cut-through connections, then only option 2 works. Additionally, you need the two apps to attach with both local and global scope set. Hope this helps!Regards, FlorinThen, my question is how to configure 192.168.1.3/24 and 192.168.1.4/24 in VPP?Thanks & Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 12/6/2019 03:56，Florin Coras wrote： Hi Hanlin, Inline.On Dec 4, 2019, at 1:59 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin，Thanks for your patient reply. Still I have some doubt inline.wanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/30/2019 02:47，Florin Coras wrote： Hi Hanlin, Inline. On Nov 29, 2019, at 7:12 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,Thanks for your reply.I just consider a very simple use case. Some apps in different containers communicate through VPP, just in a L2 bridge domain. Without hoststack, we may add some host-interfaces in one bridge domain, and assign IP address of veth interface in containers. In addition, a physical nic also added in same bridge domain to communicate with other hosts.But with hoststack, things seem complicated because we have to assign IP address inside VPP. FC: Yes, with host stack transport protocols are terminated in vpp, therefore the interfaces must have IPs. Do you need network access to the container’s linux stack for other applications, i.e., do you need IPs in the container as well? Also, can’t you give the interfaces /32 IPs?Hanlin：I need
Re: [vpp-dev] [VCL] hoststack app crash with invalid memfd segment address
That's great! I'll apply and check it soon.
Thanks & Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 12/12/2019 04:15，Florin Coras wrote：
Hi Hanlin, Thanks to Dave, we can now have per thread binary api connections to vpp. I’ve updated the socket client and vcl to leverage this so, after [1] we have per vcl worker thread binary api sockets that are used to exchange fds. Let me know if you’re still hitting the issue. Regards,Florin[1] https://gerrit.fd.io/r/c/vpp/+/23687On Nov 22, 2019, at 10:30 AM, Florin Coras <fcoras.li...@gmail.com> wrote:Hi Hanlin, Okay, that’s a different issue. The expectation is that each vcl worker has a different binary api transport into vpp. This assumption holds for applications with multiple process workers (like nginx) but is not completely satisfied for applications with thread workers. Namely, for each vcl worker we connect over the socket api to vpp and initialize the shared memory transport (so binary api messages are delivered over shared memory instead of the socket). However, as you’ve noted, the socket client is currently not multi-thread capable, consequently we have an overlap of socket client fds between the workers. The first segment is assigned properly but the subsequent ones will fail in this scenario. I wasn’t aware of this so we’ll have to either fix the socket binary api client, for multi-threaded apps, or change the session layer to use different fds for exchanging memfd fds. Regards, FlorinOn Nov 21, 2019, at 11:47 PM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,Regarding 3), I think main problem maybe in function vl_socket_client_recv_fd_msg called by vcl_session_app_add_segment_handler. Mutiple worker threads share the same scm->client_socket.fd, so B2 may receive the segment memfd belong to A1. Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/22/2019 01:44，Florin Coras wrote： Hi Hanlin, As Jon pointed out, you may want to register with gerrit. You comments with respect to points 1) and 2) are spot on. I’ve updated the patch to fix them. Regarding 3), if I understood your scenario correctly, it should not happen. The ssvm infra forces applications to map segments at fixed addresses. That is, for the scenario you’re describing lower, if B2 is processed first, ssvm_slave_init_memfd will map the segment at A2. Note how we first map the segment to read the shared header (sh) and then use sh->ssvm_va (which should be A2) to remap the segment at a fixed virtual address (va). Regards,FlorinOn Nov 21, 2019, at 2:49 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,I have applied the patch, and found some problems in my case. I have not right to post it in gerrit, so I post here.1)evt->event_type should be set with SESSION_CTRL_EVT_APP_DEL_SEGMENT rather than SESSION_CTRL_EVT_APP_ADD_SEGMENT. File: src/vnet/session/session_api.c, Line: 561, Function:mq_send_del_segment_cb2)session_send_fds may been called in the end of function mq_send_add_segment_cb, otherwise lock of app_mq can't been free here.File: src/vnet/session/session_api.c, Line: 519, Function:mq_send_add_segment_cb 3) When vcl_segment_attach called in each worker thread, then ssvm_slave_init_memfd can been called in each worker thread and then ssvm_slave_init_memfd map address sequentially through map segment once in advance. It's OK in only one thread, but maybe wrong in multiple worker threads. Suppose following scene: VPP allocate segment at address A1 and notify worker thread B1 to expect B1 also map segment at address A1, and simultaneously VPP allocate segment at address A2 and notify worker thread B2 to expect B2 map segment at address A2. If B2 first process notify message, then ssvm_slave_init_memfd may map segment at address A1. Maybe VPP can add segment map address in notify message, and then worker thread just map segment at this address. Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/19/2019 09:50，wanghanlin wrote： Hi Florin,VPP vsersion is v19.08.I'll apply this patch and check it. Thanks a lot!Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/16/2019 00:50，Florin Coras wrote： Hi Hanlin,Just to make sure, are you running m
[vpp-dev] Do we have any plan to introduce packetdrill to test VPP hoststack?
Hi All,Do we have any plan to introduce packetdrill to test VPP hoststack?Or，how do we guarantee the correct implementation of the protocol stack?Regards，Hanlin wanghanlin wanghan...@corp.netease.com 签名由 网易邮箱大师 定制 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14999): https://lists.fd.io/g/vpp-dev/message/14999 Mute This Topic: https://lists.fd.io/mt/69343227/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Do we have any plan to introduce packet =?utf-8?Q?drill_to_test_VPP_hoststack??=
Hi Florin,Okay, I got it. Thanks!Regards,Hanlin wanghanlin wanghan...@corp.netease.com 签名由 网易邮箱大师 定制 On 12/31/2019 11:56，Florin Coras wrote： Hi Hanlin,As far as I know, there are currently no plans for that, but that doesn’t mean we won’t accept contributions :-)Regarding what’s been done, over the past year, I’ve been using Defensics Codenomicon to validate TCP implementation before each release (more than 1M tests). FD.io does not own a Codenomicon license, so using something like packetdrill periodically seems like a really good alternative. Regards,FlorinOn Dec 30, 2019, at 6:52 PM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi All,Do we have any plan to introduce packetdrill to test VPP hoststack?Or，how do we guarantee the correct implementation of the protocol stack?Regards，Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制-=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.View/Reply Online (#14999): https://lists.fd.io/g/vpp-dev/message/14999Mute This Topic: https://lists.fd.io/mt/69343227/675152Group Owner: vpp-dev+ow...@lists.fd.ioUnsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]-=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15001): https://lists.fd.io/g/vpp-dev/message/15001 Mute This Topic: https://lists.fd.io/mt/69344783/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [VCL] hoststack app crash with invalid memfd segment address
Hi Florin，
With latest master code，the problem regarding 3) has been fixed.Thanks & Regards，Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 12/12/2019 14:53，wanghanlin wrote：
That's great! I'll apply and check it soon.
Thanks & Regards,Hanlin
wanghanlin
wanghan...@corp.netease.com
签名由
网易邮箱大师
定制
On 12/12/2019 04:15，Florin Coras wrote：
Hi Hanlin, Thanks to Dave, we can now have per thread binary api connections to vpp. I’ve updated the socket client and vcl to leverage this so, after [1] we have per vcl worker thread binary api sockets that are used to exchange fds. Let me know if you’re still hitting the issue. Regards,Florin[1] https://gerrit.fd.io/r/c/vpp/+/23687On Nov 22, 2019, at 10:30 AM, Florin Coras <fcoras.li...@gmail.com> wrote:Hi Hanlin, Okay, that’s a different issue. The expectation is that each vcl worker has a different binary api transport into vpp. This assumption holds for applications with multiple process workers (like nginx) but is not completely satisfied for applications with thread workers. Namely, for each vcl worker we connect over the socket api to vpp and initialize the shared memory transport (so binary api messages are delivered over shared memory instead of the socket). However, as you’ve noted, the socket client is currently not multi-thread capable, consequently we have an overlap of socket client fds between the workers. The first segment is assigned properly but the subsequent ones will fail in this scenario. I wasn’t aware of this so we’ll have to either fix the socket binary api client, for multi-threaded apps, or change the session layer to use different fds for exchanging memfd fds. Regards, FlorinOn Nov 21, 2019, at 11:47 PM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,Regarding 3), I think main problem maybe in function vl_socket_client_recv_fd_msg called by vcl_session_app_add_segment_handler. Mutiple worker threads share the same scm->client_socket.fd, so B2 may receive the segment memfd belong to A1. Regards,Hanlinwanghanlinwanghan...@corp.netease.com签名由 网易邮箱大师 定制On 11/22/2019 01:44，Florin Coras wrote： Hi Hanlin, As Jon pointed out, you may want to register with gerrit. You comments with respect to points 1) and 2) are spot on. I’ve updated the patch to fix them. Regarding 3), if I understood your scenario correctly, it should not happen. The ssvm infra forces applications to map segments at fixed addresses. That is, for the scenario you’re describing lower, if B2 is processed first, ssvm_slave_init_memfd will map the segment at A2. Note how we first map the segment to read the shared header (sh) and then use sh->ssvm_va (which should be A2) to remap the segment at a fixed virtual address (va). Regards,FlorinOn Nov 21, 2019, at 2:49 AM, wanghanlin <wanghan...@corp.netease.com> wrote:Hi Florin,I have applied the patch, and found some problems in my case. I have not right to post it in gerrit, so I post here.1)evt->event_type should be set with SESSION_CTRL_EVT_APP_DEL_SEGMENT rather than SESSION_CTRL_EVT_APP_ADD_SEGMENT. File: src/vnet/session/session_api.c, Line: 561, Function:mq_send_del_segment_cb2)session_send_fds may been called in the end of function mq_send_add_segment_cb, otherwise lock of app_mq can't been free here.File: src/vnet/session/session_api.c, Line: 519, Function:mq_send_add_segment_cb 3) When vcl_segment_attach called in each worker thread, then ssvm_slave_init_memfd can been called in each worker thread and then ssvm_slave_init_memfd map address sequentially through map segment once
