Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
If Forrest has some questions I can attend the call but we can also do it using email. At least for now it seems that when the TX queue binding and hash lookup are fixed the application could be accepted to the repo. -Matias From: Bill Fischofer [mailto:bill.fischo...@linaro.org] Sent: Monday, August 15, 2016 11:19 PM To: Elo, Matias (Nokia - FI/Espoo) Cc: forrest.shi ; lng-odp Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd Do we want to discuss this during tomorrow's ODP public call? I'll add it to the agenda. On Mon, Aug 15, 2016 at 6:36 AM, Elo, Matias (Nokia - FI/Espoo) mailto:matias@nokia-bell-labs.com>> wrote: Hi, ODP_ASSERT isn’t part of the public API. After removing it the application builds correctly. The argument parsing issue is now fixed. However, after the ODP_PKTIO_OP_MT -> ODP_PKTIO_OP_MT_UNSAFE fix the TX side is now broken. Syslog is full of error messages like: nm_txsync_prologueeth1 TX0: fail 'cur > kring->rtail && cur < head' h 399 c 394 t 307 rh 380 rc 380 rt 307 hc 380 ht 307 nm_txsync_prologueeth1 TX0: fail 'head < kring->rhead || head > kring->rtail' h 53 c 53 t 508 rh 54 rc 54 rt 508 hc 54 ht 508 netmap_ring_reinitcalled for eth1 TX0 These messages are from netmap and I would assume they are caused by using the same TX queues from multiple threads. “The rx and tx queues are linked because the qconf interface is supported, like (0,0,0),(0,1,0) The thread will receive two queues (0,1) from port 0, but no txq information is specified. My idea is let the txq_idx equals rxq_idx, and if the actual queue is not enough, using share queue.” You don’t need any additional information about the TX queues from qconf. You can always try to allocate as many TX queue as there are worker threads for each interface. In this case the txq_idx can actually be odp_thread_id() (if enough TX queues could be allocated). If the number of worker threads is larger than ‘capa.max_output_queues’ you have to use locking. -Matias From: forrest.shi [mailto:forrest@linaro.org<mailto:forrest@linaro.org>] Sent: Monday, August 15, 2016 1:18 PM To: Elo, Matias (Nokia - FI/Espoo) mailto:matias@nokia-bell-labs.com>>; Mike Holmes mailto:mike.hol...@linaro.org>> Cc: lng-odp mailto:lng-odp@lists.linaro.org>> Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd Hi Matias, The rx and tx queues are linked because the qconf interface is supported, like (0,0,0),(0,1,0) The thread will receive two queues (0,1) from port 0, but no txq information is specified. My idea is let the txq_idx equals rxq_idx, and if the actual queue is not enough, using share queue. It looks like I have mis-understood ODP_PKTIO_OP_MT_UNSAFE and ODP_PKTIO_OP_MT, and then locked txq. It will be corrected. About only running with the -d xxx, I checked the code, maybe caused by odp_barrier_wait() in if(). Here there is some assumption that the odp-linux will reserve two threads for OS and control thread. On my side, odp-nxp doesn't reserve these 2 threads, so the behavior is not the same. Sorry for confused info. About the hash, the find_fwd_db_entry should be hot. However, it's supposesd to return quickly from this function using a hash match. Have not get a clue. It is a bit slow, compared with lpm on my side. Would you please help to run the attached patch on X86? I believe the patch should fix the first 2 issues, at least on my side. Thanks, Forrest 在 8/12/2016 4:28 PM, Elo, Matias (Nokia - FI/Espoo) 写道: I tested the application on our server and the forwarding rate doesn’t scale when core count is increased (added l2fwd and OFP results for reference). Based on some debugging this seems to be caused by using locks on the TX side. ‘perf’ shows that the application spends most of time in odp_ticket_lock: 49.53% odp_l3fwd odp_l3fwd [.] odp_ticketlock_lock In the setup_worker_qconf() function’s comments you state: “Try to have one tx queue for each rx queue, if not vailable, shared tx queue is used.” You should configure one tx queue per each worker thread for all interfaces. The number of tx and rx queues shouldn’t be linked in any way. This enables you to used lockless tx. You can check how this is done in e.g. odp_l2fwd. When I change the ‘--style’ option to ‘hash’ forwarding rate drops to only a few thousand packets per second. Perf output in this case: Samples: 141K of event 'cycles:pp', Event count (approx.): 90818460609 Overhead CommandShared Object Symbol + 99.47% odp_l3fwd odp_l3fwd [.] find_fwd_db_entry -Matias From: Elo, Matias (Nokia - FI/Espoo) Sent: Friday, August 12, 2016 9:54 AM To: 'Mike Holmes' mailto:mike.hol...@linaro.org>><mailto:mike.hol...@linaro.org<mailto:mike.hol...@linaro.org>>; forrest.shi mailto:forr
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
Do we want to discuss this during tomorrow's ODP public call? I'll add it to the agenda. On Mon, Aug 15, 2016 at 6:36 AM, Elo, Matias (Nokia - FI/Espoo) < matias@nokia-bell-labs.com> wrote: > Hi, > > ODP_ASSERT isn’t part of the public API. After removing it the application > builds correctly. > > The argument parsing issue is now fixed. > > However, after the ODP_PKTIO_OP_MT -> ODP_PKTIO_OP_MT_UNSAFE fix the TX > side > is now broken. Syslog is full of error messages like: > > nm_txsync_prologueeth1 TX0: fail 'cur > kring->rtail && cur < > head' h 399 c 394 t 307 rh 380 rc 380 rt 307 hc 380 ht 307 > nm_txsync_prologueeth1 TX0: fail 'head < kring->rhead || head > > kring->rtail' h 53 c 53 t 508 rh 54 rc 54 rt 508 hc 54 ht 508 > netmap_ring_reinitcalled for eth1 TX0 > > These messages are from netmap and I would assume they are caused by using > the same > TX queues from multiple threads. > > “The rx and tx queues are linked because the qconf interface is supported, > like (0,0,0),(0,1,0) > > The thread will receive two queues (0,1) from port 0, but no txq > information is specified. > > My idea is let the txq_idx equals rxq_idx, and if the actual queue is not > enough, using share queue.” > You don’t need any additional information about the TX queues from qconf. > You can always try to > allocate as many TX queue as there are worker threads for each interface. > In this case the txq_idx can > actually be odp_thread_id() (if enough TX queues could be allocated). If > the number of worker threads > is larger than ‘capa.max_output_queues’ you have to use locking. > > -Matias > > > > From: forrest.shi [mailto:forrest@linaro.org] > Sent: Monday, August 15, 2016 1:18 PM > To: Elo, Matias (Nokia - FI/Espoo) ; Mike > Holmes > Cc: lng-odp > Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd > > > Hi Matias, > > The rx and tx queues are linked because the qconf interface is supported, > like (0,0,0),(0,1,0) > > The thread will receive two queues (0,1) from port 0, but no txq > information is specified. > > My idea is let the txq_idx equals rxq_idx, and if the actual queue is not > enough, using share queue. > > It looks like I have mis-understood ODP_PKTIO_OP_MT_UNSAFE and > ODP_PKTIO_OP_MT, and then locked txq. > > It will be corrected. > > > > About only running with the -d xxx, I checked the code, maybe caused by > odp_barrier_wait() in if(). > > Here there is some assumption that the odp-linux will reserve two threads > for OS and control thread. > > On my side, odp-nxp doesn't reserve these 2 threads, so the behavior is > not the same. Sorry for confused info. > > > > About the hash, the find_fwd_db_entry should be hot. However, it's > supposesd to return quickly from this function using a hash match. > > Have not get a clue. It is a bit slow, compared with lpm on my side. > > > > Would you please help to run the attached patch on X86? I believe the > patch should fix the first 2 issues, at least on my side. > > > > Thanks, > > Forrest > > > > 在 8/12/2016 4:28 PM, Elo, Matias (Nokia - FI/Espoo) 写道: > I tested the application on our server and the forwarding rate doesn’t > scale when core > count is increased (added l2fwd and OFP results for reference). Based on > some debugging > this seems to be caused by using locks on the TX side. > > ‘perf’ shows that the application spends most of time in odp_ticket_lock: > 49.53% odp_l3fwd odp_l3fwd [.] odp_ticketlock_lock > > In the setup_worker_qconf() function’s comments you state: “Try to have > one tx queue for > each rx queue, if not vailable, shared tx queue is used.” You should > configure one tx queue > per each worker thread for all interfaces. The number of tx and rx queues > shouldn’t be linked > in any way. This enables you to used lockless tx. You can check how this > is done in e.g. odp_l2fwd. > > When I change the ‘--style’ option to ‘hash’ forwarding rate drops to only > a few thousand packets > per second. > > Perf output in this case: > Samples: 141K of event 'cycles:pp', Event count (approx.): 90818460609 > Overhead CommandShared Object Symbol > + 99.47% odp_l3fwd odp_l3fwd [.] find_fwd_db_entry > > > -Matias > > > From: Elo, Matias (Nokia - FI/Espoo) > Sent: Friday, August 12, 2016 9:54 AM > To: 'Mike Holmes' <mailto:mike.hol...@linaro.org>; > forrest.shi <mailto:forrest@linaro.org> > Cc: lng-odp <mailto:lng-odp@lists.linaro.org> > Subject: R
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
Hi, ODP_ASSERT isn’t part of the public API. After removing it the application builds correctly. The argument parsing issue is now fixed. However, after the ODP_PKTIO_OP_MT -> ODP_PKTIO_OP_MT_UNSAFE fix the TX side is now broken. Syslog is full of error messages like: nm_txsync_prologueeth1 TX0: fail 'cur > kring->rtail && cur < head' h 399 c 394 t 307 rh 380 rc 380 rt 307 hc 380 ht 307 nm_txsync_prologueeth1 TX0: fail 'head < kring->rhead || head > kring->rtail' h 53 c 53 t 508 rh 54 rc 54 rt 508 hc 54 ht 508 netmap_ring_reinitcalled for eth1 TX0 These messages are from netmap and I would assume they are caused by using the same TX queues from multiple threads. “The rx and tx queues are linked because the qconf interface is supported, like (0,0,0),(0,1,0) The thread will receive two queues (0,1) from port 0, but no txq information is specified. My idea is let the txq_idx equals rxq_idx, and if the actual queue is not enough, using share queue.” You don’t need any additional information about the TX queues from qconf. You can always try to allocate as many TX queue as there are worker threads for each interface. In this case the txq_idx can actually be odp_thread_id() (if enough TX queues could be allocated). If the number of worker threads is larger than ‘capa.max_output_queues’ you have to use locking. -Matias From: forrest.shi [mailto:forrest@linaro.org] Sent: Monday, August 15, 2016 1:18 PM To: Elo, Matias (Nokia - FI/Espoo) ; Mike Holmes Cc: lng-odp Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd Hi Matias, The rx and tx queues are linked because the qconf interface is supported, like (0,0,0),(0,1,0) The thread will receive two queues (0,1) from port 0, but no txq information is specified. My idea is let the txq_idx equals rxq_idx, and if the actual queue is not enough, using share queue. It looks like I have mis-understood ODP_PKTIO_OP_MT_UNSAFE and ODP_PKTIO_OP_MT, and then locked txq. It will be corrected. About only running with the -d xxx, I checked the code, maybe caused by odp_barrier_wait() in if(). Here there is some assumption that the odp-linux will reserve two threads for OS and control thread. On my side, odp-nxp doesn't reserve these 2 threads, so the behavior is not the same. Sorry for confused info. About the hash, the find_fwd_db_entry should be hot. However, it's supposesd to return quickly from this function using a hash match. Have not get a clue. It is a bit slow, compared with lpm on my side. Would you please help to run the attached patch on X86? I believe the patch should fix the first 2 issues, at least on my side. Thanks, Forrest 在 8/12/2016 4:28 PM, Elo, Matias (Nokia - FI/Espoo) 写道: I tested the application on our server and the forwarding rate doesn’t scale when core count is increased (added l2fwd and OFP results for reference). Based on some debugging this seems to be caused by using locks on the TX side. ‘perf’ shows that the application spends most of time in odp_ticket_lock: 49.53% odp_l3fwd odp_l3fwd [.] odp_ticketlock_lock In the setup_worker_qconf() function’s comments you state: “Try to have one tx queue for each rx queue, if not vailable, shared tx queue is used.” You should configure one tx queue per each worker thread for all interfaces. The number of tx and rx queues shouldn’t be linked in any way. This enables you to used lockless tx. You can check how this is done in e.g. odp_l2fwd. When I change the ‘--style’ option to ‘hash’ forwarding rate drops to only a few thousand packets per second. Perf output in this case: Samples: 141K of event 'cycles:pp', Event count (approx.): 90818460609 Overhead CommandShared Object Symbol + 99.47% odp_l3fwd odp_l3fwd [.] find_fwd_db_entry -Matias From: Elo, Matias (Nokia - FI/Espoo) Sent: Friday, August 12, 2016 9:54 AM To: 'Mike Holmes' <mailto:mike.hol...@linaro.org>; forrest.shi <mailto:forrest....@linaro.org> Cc: lng-odp <mailto:lng-odp@lists.linaro.org> Subject: RE: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd Sorry for the slow review on my part. I’m currently testing the application and there seems to be something wrong with the argument parsing. With the following commands no packets are forwarded. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 0 or $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 Only when some ‘-d’ value is given packets are actually forwarded E.g. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 60 -Matias From: Mike Holmes [mailto:mike.hol...@linaro.org] Sent: Thursday, August 11, 2016 7:20 PM To: forrest.shi mailto:forrest@linaro.org>> Cc: lng-odp mailto:lng-odp@lists.linaro.org>>; Elo, Matias (Nokia - FI/Espoo) ma
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
Hi Forrest, No packets are forwarded unless the ‘-d’ option with a value greater than zero is given. I’m using an external Ixia traffic generator. E.g. No packets forwarded: sudo ./example/l3fwd/odp_l3fwd -i p2p1,p1p1 -r 1.1.1.0/24,p2p1 -r 2.2.2.0/24,p1p1 -t 1 Packets forwarded: sudo ./example/l3fwd/odp_l3fwd -i p2p1,p1p1 -r 1.1.1.0/24,p2p1 -r 2.2.2.0/24,p1p1 -t 1 -d 999 -Matias From: forrest.shi [mailto:forrest@linaro.org] Sent: Monday, August 15, 2016 8:17 AM To: Elo, Matias (Nokia - FI/Espoo) ; Mike Holmes Cc: lng-odp Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd Hi Matias, Many thanks for your review and testing. "With the following commands no packets are forwarded." Do you mean really no packcet forwarding, or just no output like below? x pps, x max pps, 0 rx drops, 0 tx drops If you are meaning the output, this is the behavior expected. -d 0 equals without -d, no output print, but forwarding is really working forever. -d x x>0, the program will run x seconds and print the output every 10s. At my side, the program runs as I said above. thanks, Forrest 在 8/12/2016 2:54 PM, Elo, Matias (Nokia - FI/Espoo) 写道: Sorry for the slow review on my part. I’m currently testing the application and there seems to be something wrong with the argument parsing. With the following commands no packets are forwarded. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 0 or $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 Only when some ‘-d’ value is given packets are actually forwarded E.g. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 60 -Matias From: Mike Holmes [mailto:mike.hol...@linaro.org] Sent: Thursday, August 11, 2016 7:20 PM To: forrest.shi <mailto:forrest@linaro.org> Cc: lng-odp <mailto:lng-odp@lists.linaro.org>; Elo, Matias (Nokia - FI/Espoo) <mailto:matias@nokia-bell-labs.com>; Savolainen, Petri (Nokia - FI/Espoo) <mailto:petri.savolai...@nokia-bell-labs.com> Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd I CC'ed Matias, I am not sure if we can get another Member to run this on their HW ? It does not have to be perfect but I think it is likely to be ready as an example at this point. Mike On 10 August 2016 at 06:42, forrest.shi mailto:forrest@linaro.org>> wrote: Ping, need review and comments. Pending a bit long. > -Original Message- > From: forrest@linaro.org<mailto:forrest@linaro.org> > [mailto:forrest@linaro.org<mailto:forrest@linaro.org>] > Sent: Thursday, July 28, 2016 16:52 > To: lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> > Cc: Xuelin Shi mailto:forrest@linaro.org>> > Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd > > From: Xuelin Shi mailto:forrest@linaro.org>> > > multi-thread, multi-queues and bi-directional forwarding. > > support (port, queue, thread) arguments in cmdline which specify how > the threads handle which rx queue at which port, if no this argument, > default specification used. > > both hash and lpm based lookup methods are supported, default lpm. > > Signed-off-by: Xuelin Shi > mailto:forrest@linaro.org>> > --- > example/Makefile.am | 12 +- > example/l3fwd/.gitignore |1 + > example/l3fwd/Makefile.am | 12 + > example/l3fwd/odp_l3fwd.c | 1094 > + > example/l3fwd/odp_l3fwd_db.c | 432 > example/l3fwd/odp_l3fwd_db.h | 136 + > example/l3fwd/odp_l3fwd_lpm.c | 224 + > example/l3fwd/odp_l3fwd_lpm.h | 20 + > example/m4/configure.m4 |9 +- > 9 files changed, 1935 insertions(+), 5 deletions(-) > create mode 100644 example/l3fwd/.gitignore > create mode 100644 example/l3fwd/Makefile.am > create mode 100644 example/l3fwd/odp_l3fwd.c > create mode 100644 example/l3fwd/odp_l3fwd_db.c > create mode 100644 example/l3fwd/odp_l3fwd_db.h > create mode 100644 example/l3fwd/odp_l3fwd_lpm.c > create mode 100644 example/l3fwd/odp_l3fwd_lpm.h > > diff --git a/example/Makefile.am b/example/Makefile.am > index 37542af..dfc07b6 100644 > --- a/example/Makefile.am > +++ b/example/Makefile.am > @@ -1 +1,11 @@ > -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt > l2fwd_simple switch hello > +SUBDIRS = classifier \ > + generator \ > + hello \ > + ipsec \ > + l2fwd_simple \ > + l3fwd \ > + packet \ > + switch \ > + time \ > + timer \ > + traffic_mgmt > diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore > new file mode 100644 > index 000..74e501f > --- /dev/null &
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
Hi Matias, Many thanks for your review and testing. "With the following commands no packets are forwarded." Do you mean really no packcet forwarding, or just no output like below? x pps, x max pps, 0 rx drops, 0 tx drops If you are meaning the output, this is the behavior expected. -d 0 equals without -d, no output print, but forwarding is really working forever. -d x x>0, the program will run x seconds and print the output every 10s. At my side, the program runs as I said above. thanks, Forrest 在 8/12/2016 2:54 PM, Elo, Matias (Nokia - FI/Espoo) 写道: Sorry for the slow review on my part. I’m currently testing the application and there seems to be something wrong with the argument parsing. With the following commands no packets are forwarded. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 0 or $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 Only when some ‘-d’ value is given packets are actually forwarded E.g. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 60 -Matias *From:*Mike Holmes [mailto:mike.hol...@linaro.org] *Sent:* Thursday, August 11, 2016 7:20 PM *To:* forrest.shi *Cc:* lng-odp ; Elo, Matias (Nokia - FI/Espoo) ; Savolainen, Petri (Nokia - FI/Espoo) *Subject:* Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd I CC'ed Matias, I am not sure if we can get another Member to run this on their HW ? It does not have to be perfect but I think it is likely to be ready as an example at this point. Mike On 10 August 2016 at 06:42, forrest.shi <mailto:forrest@linaro.org>> wrote: Ping, need review and comments. Pending a bit long. > -Original Message- > From: forrest@linaro.org <mailto:forrest@linaro.org> [mailto:forrest@linaro.org <mailto:forrest@linaro.org>] > Sent: Thursday, July 28, 2016 16:52 > To: lng-odp@lists.linaro.org <mailto:lng-odp@lists.linaro.org> > Cc: Xuelin Shi mailto:forrest....@linaro.org>> > Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd > > From: Xuelin Shi mailto:forrest@linaro.org>> > > multi-thread, multi-queues and bi-directional forwarding. > > support (port, queue, thread) arguments in cmdline which specify how > the threads handle which rx queue at which port, if no this argument, > default specification used. > > both hash and lpm based lookup methods are supported, default lpm. > > Signed-off-by: Xuelin Shi mailto:forrest@linaro.org>> > --- > example/Makefile.am | 12 +- > example/l3fwd/.gitignore |1 + > example/l3fwd/Makefile.am | 12 + > example/l3fwd/odp_l3fwd.c | 1094 > + > example/l3fwd/odp_l3fwd_db.c | 432 > example/l3fwd/odp_l3fwd_db.h | 136 + > example/l3fwd/odp_l3fwd_lpm.c | 224 + > example/l3fwd/odp_l3fwd_lpm.h | 20 + > example/m4/configure.m4 |9 +- > 9 files changed, 1935 insertions(+), 5 deletions(-) > create mode 100644 example/l3fwd/.gitignore > create mode 100644 example/l3fwd/Makefile.am > create mode 100644 example/l3fwd/odp_l3fwd.c > create mode 100644 example/l3fwd/odp_l3fwd_db.c > create mode 100644 example/l3fwd/odp_l3fwd_db.h > create mode 100644 example/l3fwd/odp_l3fwd_lpm.c > create mode 100644 example/l3fwd/odp_l3fwd_lpm.h > > diff --git a/example/Makefile.am b/example/Makefile.am > index 37542af..dfc07b6 100644 > --- a/example/Makefile.am > +++ b/example/Makefile.am > @@ -1 +1,11 @@ > -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt > l2fwd_simple switch hello > +SUBDIRS = classifier \ > + generator \ > + hello \ > + ipsec \ > + l2fwd_simple \ > + l3fwd \ > + packet \ > + switch \ > + time \ > + timer \ > + traffic_mgmt > diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore > new file mode 100644 > index 000..74e501f > --- /dev/null > +++ b/example/l3fwd/.gitignore > @@ -0,0 +1 @@ > +odp_l3fwd > diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am > new file mode 100644 > index 000..f926717 > --- /dev/null > +++ b/example/l3fwd/Makefile.am > @@ -0,0 +1,12 @@ > +include $(top_srcdir)/example/Makefile.inc > + > +bin_PROGRAMS = odp_l3fwd$(EXEEXT) > +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static > +odp_l3fwd_CFLAGS = $(AM_CF
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
I tested the application on our server and the forwarding rate doesn’t scale when core count is increased (added l2fwd and OFP results for reference). Based on some debugging this seems to be caused by using locks on the TX side. ‘perf’ shows that the application spends most of time in odp_ticket_lock: 49.53% odp_l3fwd odp_l3fwd [.] odp_ticketlock_lock In the setup_worker_qconf() function’s comments you state: “Try to have one tx queue for each rx queue, if not vailable, shared tx queue is used.” You should configure one tx queue per each worker thread for all interfaces. The number of tx and rx queues shouldn’t be linked in any way. This enables you to used lockless tx. You can check how this is done in e.g. odp_l2fwd. When I change the ‘--style’ option to ‘hash’ forwarding rate drops to only a few thousand packets per second. Perf output in this case: Samples: 141K of event 'cycles:pp', Event count (approx.): 90818460609 Overhead CommandShared Object Symbol + 99.47% odp_l3fwd odp_l3fwd [.] find_fwd_db_entry -Matias From: Elo, Matias (Nokia - FI/Espoo) Sent: Friday, August 12, 2016 9:54 AM To: 'Mike Holmes' ; forrest.shi Cc: lng-odp Subject: RE: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd Sorry for the slow review on my part. I’m currently testing the application and there seems to be something wrong with the argument parsing. With the following commands no packets are forwarded. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 0 or $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 Only when some ‘-d’ value is given packets are actually forwarded E.g. $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 60 -Matias From: Mike Holmes [mailto:mike.hol...@linaro.org] Sent: Thursday, August 11, 2016 7:20 PM To: forrest.shi mailto:forrest@linaro.org>> Cc: lng-odp mailto:lng-odp@lists.linaro.org>>; Elo, Matias (Nokia - FI/Espoo) mailto:matias@nokia-bell-labs.com>>; Savolainen, Petri (Nokia - FI/Espoo) mailto:petri.savolai...@nokia-bell-labs.com>> Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd I CC'ed Matias, I am not sure if we can get another Member to run this on their HW ? It does not have to be perfect but I think it is likely to be ready as an example at this point. Mike On 10 August 2016 at 06:42, forrest.shi mailto:forrest@linaro.org>> wrote: Ping, need review and comments. Pending a bit long. > -Original Message- > From: forrest@linaro.org<mailto:forrest@linaro.org> > [mailto:forrest@linaro.org<mailto:forrest@linaro.org>] > Sent: Thursday, July 28, 2016 16:52 > To: lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> > Cc: Xuelin Shi mailto:forrest@linaro.org>> > Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd > > From: Xuelin Shi mailto:forrest@linaro.org>> > > multi-thread, multi-queues and bi-directional forwarding. > > support (port, queue, thread) arguments in cmdline which specify how > the threads handle which rx queue at which port, if no this argument, > default specification used. > > both hash and lpm based lookup methods are supported, default lpm. > > Signed-off-by: Xuelin Shi > mailto:forrest@linaro.org>> > --- > example/Makefile.am | 12 +- > example/l3fwd/.gitignore |1 + > example/l3fwd/Makefile.am | 12 + > example/l3fwd/odp_l3fwd.c | 1094 > + > example/l3fwd/odp_l3fwd_db.c | 432 > example/l3fwd/odp_l3fwd_db.h | 136 + > example/l3fwd/odp_l3fwd_lpm.c | 224 + > example/l3fwd/odp_l3fwd_lpm.h | 20 + > example/m4/configure.m4 |9 +- > 9 files changed, 1935 insertions(+), 5 deletions(-) > create mode 100644 example/l3fwd/.gitignore > create mode 100644 example/l3fwd/Makefile.am > create mode 100644 example/l3fwd/odp_l3fwd.c > create mode 100644 example/l3fwd/odp_l3fwd_db.c > create mode 100644 example/l3fwd/odp_l3fwd_db.h > create mode 100644 example/l3fwd/odp_l3fwd_lpm.c > create mode 100644 example/l3fwd/odp_l3fwd_lpm.h > > diff --git a/example/Makefile.am b/example/Makefile.am > index 37542af..dfc07b6 100644 > --- a/example/Makefile.am > +++ b/example/Makefile.am > @@ -1 +1,11 @@ > -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt > l2fwd_simple switch hello > +SUBDIRS = classifier \ > + generator \ > + hello \ > + ipsec \ > + l2fwd_simple \ > + l3fwd \ > + packet \ > + switch \ > + time \ > + timer \ > + traffic_mgmt > diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
Sorry for the slow review on my part. I’m currently testing the application and
there seems to be something wrong with the argument parsing.
With the following commands no packets are forwarded.
$ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d 0
or
$ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2
Only when some ‘-d’ value is given packets are actually forwarded
E.g.
$ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 -t 2 -d
60
-Matias
From: Mike Holmes [mailto:mike.hol...@linaro.org]
Sent: Thursday, August 11, 2016 7:20 PM
To: forrest.shi
Cc: lng-odp ; Elo, Matias (Nokia - FI/Espoo)
; Savolainen, Petri (Nokia - FI/Espoo)
Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
I CC'ed Matias, I am not sure if we can get another Member to run this on their
HW ?
It does not have to be perfect but I think it is likely to be ready as an
example at this point.
Mike
On 10 August 2016 at 06:42, forrest.shi
mailto:forrest@linaro.org>> wrote:
Ping, need review and comments. Pending a bit long.
> -Original Message-
> From: forrest@linaro.org<mailto:forrest@linaro.org>
> [mailto:forrest@linaro.org<mailto:forrest@linaro.org>]
> Sent: Thursday, July 28, 2016 16:52
> To: lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org>
> Cc: Xuelin Shi mailto:forrest....@linaro.org>>
> Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd
>
> From: Xuelin Shi mailto:forrest@linaro.org>>
>
> multi-thread, multi-queues and bi-directional forwarding.
>
> support (port, queue, thread) arguments in cmdline which specify how
> the threads handle which rx queue at which port, if no this argument,
> default specification used.
>
> both hash and lpm based lookup methods are supported, default lpm.
>
> Signed-off-by: Xuelin Shi
> mailto:forrest@linaro.org>>
> ---
> example/Makefile.am | 12 +-
> example/l3fwd/.gitignore |1 +
> example/l3fwd/Makefile.am | 12 +
> example/l3fwd/odp_l3fwd.c | 1094
> +
> example/l3fwd/odp_l3fwd_db.c | 432
> example/l3fwd/odp_l3fwd_db.h | 136 +
> example/l3fwd/odp_l3fwd_lpm.c | 224 +
> example/l3fwd/odp_l3fwd_lpm.h | 20 +
> example/m4/configure.m4 |9 +-
> 9 files changed, 1935 insertions(+), 5 deletions(-)
> create mode 100644 example/l3fwd/.gitignore
> create mode 100644 example/l3fwd/Makefile.am
> create mode 100644 example/l3fwd/odp_l3fwd.c
> create mode 100644 example/l3fwd/odp_l3fwd_db.c
> create mode 100644 example/l3fwd/odp_l3fwd_db.h
> create mode 100644 example/l3fwd/odp_l3fwd_lpm.c
> create mode 100644 example/l3fwd/odp_l3fwd_lpm.h
>
> diff --git a/example/Makefile.am b/example/Makefile.am
> index 37542af..dfc07b6 100644
> --- a/example/Makefile.am
> +++ b/example/Makefile.am
> @@ -1 +1,11 @@
> -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt
> l2fwd_simple switch hello
> +SUBDIRS = classifier \
> + generator \
> + hello \
> + ipsec \
> + l2fwd_simple \
> + l3fwd \
> + packet \
> + switch \
> + time \
> + timer \
> + traffic_mgmt
> diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore
> new file mode 100644
> index 000..74e501f
> --- /dev/null
> +++ b/example/l3fwd/.gitignore
> @@ -0,0 +1 @@
> +odp_l3fwd
> diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am
> new file mode 100644
> index 000..f926717
> --- /dev/null
> +++ b/example/l3fwd/Makefile.am
> @@ -0,0 +1,12 @@
> +include $(top_srcdir)/example/Makefile.inc
> +
> +bin_PROGRAMS = odp_l3fwd$(EXEEXT)
> +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static
> +odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -
> I${top_srcdir}/test
> +
> +noinst_HEADERS = \
> + $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \
> + $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \
> + $(top_srcdir)/example/example_debug.h
> +
> +dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c
> diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c
> new file mode 100644
> index 000..b1db002
> --- /dev/null
> +++ b/example/l3fwd/odp_l3fwd.c
> @@ -0,0 +1,1094 @@
> +/* Copyright (c) 2016, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#include
> +
> +#include
> +#include
> +#include
> +#include
> +#inclu
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
I CC'ed Matias, I am not sure if we can get another Member to run this on
their HW ?
It does not have to be perfect but I think it is likely to be ready as an
example at this point.
Mike
On 10 August 2016 at 06:42, forrest.shi wrote:
> Ping, need review and comments. Pending a bit long.
>
> > -Original Message-
> > From: forrest@linaro.org [mailto:forrest@linaro.org]
> > Sent: Thursday, July 28, 2016 16:52
> > To: lng-odp@lists.linaro.org
> > Cc: Xuelin Shi
> > Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd
> >
> > From: Xuelin Shi
> >
> > multi-thread, multi-queues and bi-directional forwarding.
> >
> > support (port, queue, thread) arguments in cmdline which specify how
> > the threads handle which rx queue at which port, if no this argument,
> > default specification used.
> >
> > both hash and lpm based lookup methods are supported, default lpm.
> >
> > Signed-off-by: Xuelin Shi
> > ---
> > example/Makefile.am | 12 +-
> > example/l3fwd/.gitignore |1 +
> > example/l3fwd/Makefile.am | 12 +
> > example/l3fwd/odp_l3fwd.c | 1094
> > +
> > example/l3fwd/odp_l3fwd_db.c | 432
> > example/l3fwd/odp_l3fwd_db.h | 136 +
> > example/l3fwd/odp_l3fwd_lpm.c | 224 +
> > example/l3fwd/odp_l3fwd_lpm.h | 20 +
> > example/m4/configure.m4 |9 +-
> > 9 files changed, 1935 insertions(+), 5 deletions(-)
> > create mode 100644 example/l3fwd/.gitignore
> > create mode 100644 example/l3fwd/Makefile.am
> > create mode 100644 example/l3fwd/odp_l3fwd.c
> > create mode 100644 example/l3fwd/odp_l3fwd_db.c
> > create mode 100644 example/l3fwd/odp_l3fwd_db.h
> > create mode 100644 example/l3fwd/odp_l3fwd_lpm.c
> > create mode 100644 example/l3fwd/odp_l3fwd_lpm.h
> >
> > diff --git a/example/Makefile.am b/example/Makefile.am
> > index 37542af..dfc07b6 100644
> > --- a/example/Makefile.am
> > +++ b/example/Makefile.am
> > @@ -1 +1,11 @@
> > -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt
> > l2fwd_simple switch hello
> > +SUBDIRS = classifier \
> > + generator \
> > + hello \
> > + ipsec \
> > + l2fwd_simple \
> > + l3fwd \
> > + packet \
> > + switch \
> > + time \
> > + timer \
> > + traffic_mgmt
> > diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore
> > new file mode 100644
> > index 000..74e501f
> > --- /dev/null
> > +++ b/example/l3fwd/.gitignore
> > @@ -0,0 +1 @@
> > +odp_l3fwd
> > diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am
> > new file mode 100644
> > index 000..f926717
> > --- /dev/null
> > +++ b/example/l3fwd/Makefile.am
> > @@ -0,0 +1,12 @@
> > +include $(top_srcdir)/example/Makefile.inc
> > +
> > +bin_PROGRAMS = odp_l3fwd$(EXEEXT)
> > +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static
> > +odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -
> > I${top_srcdir}/test
> > +
> > +noinst_HEADERS = \
> > + $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \
> > + $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \
> > + $(top_srcdir)/example/example_debug.h
> > +
> > +dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c
> > diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c
> > new file mode 100644
> > index 000..b1db002
> > --- /dev/null
> > +++ b/example/l3fwd/odp_l3fwd.c
> > @@ -0,0 +1,1094 @@
> > +/* Copyright (c) 2016, Linaro Limited
> > + * All rights reserved.
> > + *
> > + * SPDX-License-Identifier: BSD-3-Clause
> > + */
> > +
> > +#include
> > +#include
> > +#include
> > +#include
> > +#include
> > +#include
> > +
> > +#include
> > +
> > +#include
> > +#include
> > +#include
> > +#include
> > +#include
> > +#include
> > +
> > +#include "odp_l3fwd_db.h"
> > +#include "odp_l3fwd_lpm.h"
> > +
> > +#define POOL_NUM_PKT 8192
> > +#define POOL_SEG_LEN 1856
> > +#define MAX_PKT_BURST32
> > +
> > +#define MAX_NB_WORKER32
> > +#define MAX_NB_PKTIO 32
> > +#define MAX_NB_QUEUE 32
> > +#define MAX_NB_QCONFS1024
> > +#define MAX_NB_ROUTE 32
> > +
> > +#define INVALID_ID (-1)
>
Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd
Ping, need review and comments. Pending a bit long.
> -Original Message-
> From: forrest@linaro.org [mailto:forrest@linaro.org]
> Sent: Thursday, July 28, 2016 16:52
> To: lng-odp@lists.linaro.org
> Cc: Xuelin Shi
> Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd
>
> From: Xuelin Shi
>
> multi-thread, multi-queues and bi-directional forwarding.
>
> support (port, queue, thread) arguments in cmdline which specify how
> the threads handle which rx queue at which port, if no this argument,
> default specification used.
>
> both hash and lpm based lookup methods are supported, default lpm.
>
> Signed-off-by: Xuelin Shi
> ---
> example/Makefile.am | 12 +-
> example/l3fwd/.gitignore |1 +
> example/l3fwd/Makefile.am | 12 +
> example/l3fwd/odp_l3fwd.c | 1094
> +
> example/l3fwd/odp_l3fwd_db.c | 432
> example/l3fwd/odp_l3fwd_db.h | 136 +
> example/l3fwd/odp_l3fwd_lpm.c | 224 +
> example/l3fwd/odp_l3fwd_lpm.h | 20 +
> example/m4/configure.m4 |9 +-
> 9 files changed, 1935 insertions(+), 5 deletions(-)
> create mode 100644 example/l3fwd/.gitignore
> create mode 100644 example/l3fwd/Makefile.am
> create mode 100644 example/l3fwd/odp_l3fwd.c
> create mode 100644 example/l3fwd/odp_l3fwd_db.c
> create mode 100644 example/l3fwd/odp_l3fwd_db.h
> create mode 100644 example/l3fwd/odp_l3fwd_lpm.c
> create mode 100644 example/l3fwd/odp_l3fwd_lpm.h
>
> diff --git a/example/Makefile.am b/example/Makefile.am
> index 37542af..dfc07b6 100644
> --- a/example/Makefile.am
> +++ b/example/Makefile.am
> @@ -1 +1,11 @@
> -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt
> l2fwd_simple switch hello
> +SUBDIRS = classifier \
> + generator \
> + hello \
> + ipsec \
> + l2fwd_simple \
> + l3fwd \
> + packet \
> + switch \
> + time \
> + timer \
> + traffic_mgmt
> diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore
> new file mode 100644
> index 000..74e501f
> --- /dev/null
> +++ b/example/l3fwd/.gitignore
> @@ -0,0 +1 @@
> +odp_l3fwd
> diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am
> new file mode 100644
> index 000..f926717
> --- /dev/null
> +++ b/example/l3fwd/Makefile.am
> @@ -0,0 +1,12 @@
> +include $(top_srcdir)/example/Makefile.inc
> +
> +bin_PROGRAMS = odp_l3fwd$(EXEEXT)
> +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static
> +odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -
> I${top_srcdir}/test
> +
> +noinst_HEADERS = \
> + $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \
> + $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \
> + $(top_srcdir)/example/example_debug.h
> +
> +dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c
> diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c
> new file mode 100644
> index 000..b1db002
> --- /dev/null
> +++ b/example/l3fwd/odp_l3fwd.c
> @@ -0,0 +1,1094 @@
> +/* Copyright (c) 2016, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#include
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#include "odp_l3fwd_db.h"
> +#include "odp_l3fwd_lpm.h"
> +
> +#define POOL_NUM_PKT 8192
> +#define POOL_SEG_LEN 1856
> +#define MAX_PKT_BURST32
> +
> +#define MAX_NB_WORKER32
> +#define MAX_NB_PKTIO 32
> +#define MAX_NB_QUEUE 32
> +#define MAX_NB_QCONFS1024
> +#define MAX_NB_ROUTE 32
> +
> +#define INVALID_ID (-1)
> +#define PRINT_INTERVAL 10 /* interval seconds of printing stats
*/
> +
> +/** Get rid of path in filename - only for unix-type paths using '/' */
> +#define NO_PATH(file_name) (strrchr((file_name), '/') ? \
> + strrchr((file_name), '/') + 1 : (file_name))
> +
> +struct l3fwd_pktio_s {
> + odp_pktio_t pktio;
> + odph_ethaddr_t mac_addr;
> + odp_pktin_queue_t ifin[MAX_NB_QUEUE];
> + odp_pktout_queue_t ifout[MAX_NB_QUEUE];
> + int nb_rxq;
> + int nb_txq;
> + int rxq_idx;
> + int txq_idx;
> +};
> +
> +struct l3fwd_qconf_s {
> + uint8_t if_idx; /* port index */
> + uint8_t rxq_idx;/* recv queue index in a port */
> + uint8_t core_idx; /* this core should handle traff
[lng-odp] [PATCHv7 1/2] example: introducing l3fwd
From: Xuelin Shi
multi-thread, multi-queues and bi-directional forwarding.
support (port, queue, thread) arguments in cmdline which specify how
the threads handle which rx queue at which port, if no this argument,
default specification used.
both hash and lpm based lookup methods are supported, default lpm.
Signed-off-by: Xuelin Shi
---
example/Makefile.am | 12 +-
example/l3fwd/.gitignore |1 +
example/l3fwd/Makefile.am | 12 +
example/l3fwd/odp_l3fwd.c | 1094 +
example/l3fwd/odp_l3fwd_db.c | 432
example/l3fwd/odp_l3fwd_db.h | 136 +
example/l3fwd/odp_l3fwd_lpm.c | 224 +
example/l3fwd/odp_l3fwd_lpm.h | 20 +
example/m4/configure.m4 |9 +-
9 files changed, 1935 insertions(+), 5 deletions(-)
create mode 100644 example/l3fwd/.gitignore
create mode 100644 example/l3fwd/Makefile.am
create mode 100644 example/l3fwd/odp_l3fwd.c
create mode 100644 example/l3fwd/odp_l3fwd_db.c
create mode 100644 example/l3fwd/odp_l3fwd_db.h
create mode 100644 example/l3fwd/odp_l3fwd_lpm.c
create mode 100644 example/l3fwd/odp_l3fwd_lpm.h
diff --git a/example/Makefile.am b/example/Makefile.am
index 37542af..dfc07b6 100644
--- a/example/Makefile.am
+++ b/example/Makefile.am
@@ -1 +1,11 @@
-SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt
l2fwd_simple switch hello
+SUBDIRS = classifier \
+ generator \
+ hello \
+ ipsec \
+ l2fwd_simple \
+ l3fwd \
+ packet \
+ switch \
+ time \
+ timer \
+ traffic_mgmt
diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore
new file mode 100644
index 000..74e501f
--- /dev/null
+++ b/example/l3fwd/.gitignore
@@ -0,0 +1 @@
+odp_l3fwd
diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am
new file mode 100644
index 000..f926717
--- /dev/null
+++ b/example/l3fwd/Makefile.am
@@ -0,0 +1,12 @@
+include $(top_srcdir)/example/Makefile.inc
+
+bin_PROGRAMS = odp_l3fwd$(EXEEXT)
+odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static
+odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -I${top_srcdir}/test
+
+noinst_HEADERS = \
+ $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \
+ $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \
+ $(top_srcdir)/example/example_debug.h
+
+dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c
diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c
new file mode 100644
index 000..b1db002
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd.c
@@ -0,0 +1,1094 @@
+/* Copyright (c) 2016, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "odp_l3fwd_db.h"
+#include "odp_l3fwd_lpm.h"
+
+#define POOL_NUM_PKT 8192
+#define POOL_SEG_LEN 1856
+#define MAX_PKT_BURST 32
+
+#define MAX_NB_WORKER 32
+#define MAX_NB_PKTIO 32
+#define MAX_NB_QUEUE 32
+#define MAX_NB_QCONFS 1024
+#define MAX_NB_ROUTE 32
+
+#define INVALID_ID (-1)
+#define PRINT_INTERVAL 10 /* interval seconds of printing stats */
+
+/** Get rid of path in filename - only for unix-type paths using '/' */
+#define NO_PATH(file_name) (strrchr((file_name), '/') ? \
+ strrchr((file_name), '/') + 1 : (file_name))
+
+struct l3fwd_pktio_s {
+ odp_pktio_t pktio;
+ odph_ethaddr_t mac_addr;
+ odp_pktin_queue_t ifin[MAX_NB_QUEUE];
+ odp_pktout_queue_t ifout[MAX_NB_QUEUE];
+ int nb_rxq;
+ int nb_txq;
+ int rxq_idx;
+ int txq_idx;
+};
+
+struct l3fwd_qconf_s {
+ uint8_t if_idx; /* port index */
+ uint8_t rxq_idx;/* recv queue index in a port */
+ uint8_t core_idx; /* this core should handle traffic */
+};
+
+struct thread_arg_s {
+ uint64_t packets;
+ uint64_t rx_drops;
+ uint64_t tx_drops;
+ struct {
+ int used;
+ int rxq[MAX_NB_QUEUE];
+ int txq[MAX_NB_QUEUE];
+ } pktio[MAX_NB_PKTIO];
+ int thr_idx;
+ int nb_pktio;
+};
+
+typedef struct {
+ char *if_names[MAX_NB_PKTIO];
+ int if_count;
+ char *route_str[MAX_NB_ROUTE];
+ int worker_count;
+ struct l3fwd_qconf_s qconf_config[MAX_NB_QCONFS];
+ int qconf_count;
+ uint32_t duration; /* seconds to run */
+ uint8_t hash_mode; /* 1:hash, 0:lpm */
+ uint8_t dest_mac_changed[MAX_NB_PKTIO]; /* 1: dest mac from cmdline */
+} app_args_t;
+
+struct {
+ app_args_t cmd_args;
+ struct l3fwd_pktio_sl3fwd_pktios[MAX_NB_PKTIO];
+ odph_odpthread_tl3fwd_workers[MAX_NB_WORKER];
+ struct thread_arg_s worker_args[MAX_NB_WORKER];
+ odph_ethaddr_t eth_dest_mac[MAX_NB_PKTIO];
+
