[dpdk-dev] DPDK RSS support for ixgbevf PMD
Hi, I'm running an ixgbevf PMD on an AWS guests with extended networking (SR-IOV functions of 82599 Intel's NIC) and noticed that even in the current git tree there is no support for a multi-queue in this PMD: reta size returned by rte_eth_dev_info_get() call is 0, while max_rx_queues and max_tx_queues are both 4. Linux ixgbevf-2.15.3 driver on the other hand successfully initializes 2 RSS queues: for some reason it always limits the number of RSS queues by 2. ixgbevf_main.c: line 2539 u16 rss = min_t(u16, num_online_cpus(), 2); The above is strange since if MRQE is set to 1010b there are 4 RSS queues available which seems to be the case in my AWS Guest. However, let's get back to DPDK. As I've mentioned above the SR-IOV function i have is RSS capable (to be 100% sure I've verified both queues are receiving packets in a multi-socket TCP test). And it's a shame I can't utilize it with a DPDK. I wonder if there are any blockers to add this capability to the ixgbevf PMD and if not is it scheduled to some time soon? Thanks in advance, vlad
[dpdk-dev] DPDK RSS support for ixgbevf PMD
2014-12-15 21:11, Vladislav Zolotarov: > Hi, > I'm running an ixgbevf PMD on an AWS guests with extended networking > (SR-IOV functions of 82599 Intel's NIC) and noticed that even in the > current git tree there is no support for a multi-queue in this PMD: reta > size returned by rte_eth_dev_info_get() call is 0, while max_rx_queues and > max_tx_queues are both 4. > > Linux ixgbevf-2.15.3 driver on the other hand successfully initializes 2 > RSS queues: for some reason it always limits the number of RSS queues by 2. > > ixgbevf_main.c: line 2539 > u16 rss = min_t(u16, num_online_cpus(), 2); > > The above is strange since if MRQE is set to 1010b there are 4 RSS queues > available which seems to be the case in my AWS Guest. > > However, let's get back to DPDK. As I've mentioned above the SR-IOV > function i have is RSS capable (to be 100% sure I've verified both queues > are receiving packets in a multi-socket TCP test). And it's a shame I can't > utilize it with a DPDK. Yes, it is not yet supported. But a patch was recently sent: http://dpdk.org/ml/archives/dev/2014-December/010028.html > I wonder if there are any blockers to add this capability to the ixgbevf > PMD and if not is it scheduled to some time soon? Please jump in the above thread for your questions. My turn to ask a question: Does it mean DPDK is going to be supported in your system (OSv)? I've seen it in the OSv roadmap. Could we have more details about how it would be integrated? Thanks -- Thomas
[dpdk-dev] DPDK RSS support for ixgbevf PMD
On Mon, Dec 15, 2014 at 10:33 PM, Thomas Monjalon wrote: > > 2014-12-15 21:11, Vladislav Zolotarov: > > Hi, > > I'm running an ixgbevf PMD on an AWS guests with extended networking > > (SR-IOV functions of 82599 Intel's NIC) and noticed that even in the > > current git tree there is no support for a multi-queue in this PMD: reta > > size returned by rte_eth_dev_info_get() call is 0, while max_rx_queues > and > > max_tx_queues are both 4. > > > > Linux ixgbevf-2.15.3 driver on the other hand successfully initializes 2 > > RSS queues: for some reason it always limits the number of RSS queues by > 2. > > > > ixgbevf_main.c: line 2539 > > u16 rss = min_t(u16, num_online_cpus(), 2); > > > > The above is strange since if MRQE is set to 1010b there are 4 RSS queues > > available which seems to be the case in my AWS Guest. > > > > However, let's get back to DPDK. As I've mentioned above the SR-IOV > > function i have is RSS capable (to be 100% sure I've verified both queues > > are receiving packets in a multi-socket TCP test). And it's a shame I > can't > > utilize it with a DPDK. > > Yes, it is not yet supported. > But a patch was recently sent: > http://dpdk.org/ml/archives/dev/2014-December/010028.html Good to know and nice timing > > t > > I wonder if there are any blockers to add this capability to the ixgbevf > > PMD and if not is it scheduled to some time soon? > > Please jump in the above thread for your questions. > > My turn to ask a question: > Does it mean DPDK is going to be supported in your system (OSv)? > In general yes. It's premature to ask now since there are components that already implemented in OSv and won't be needed (efficient producer-consumer, huge pages, etc). We sure like to reuse the driver code, it's a pity not to contribute to the same code base. We'll certainly like to let unmodified dpdk apps run as well Initially we're playing w/ the Linux dpdk on the host and AWS to gain familiarity w/ it > I've seen it in the OSv roadmap. > Could we have more details about how it would be integrated? > > Thanks > -- > Thomas >
[dpdk-dev] DPDK RSS support for ixgbevf PMD
On 12/15/14 22:33, Thomas Monjalon wrote: > 2014-12-15 21:11, Vladislav Zolotarov: >> Hi, >> I'm running an ixgbevf PMD on an AWS guests with extended networking >> (SR-IOV functions of 82599 Intel's NIC) and noticed that even in the >> current git tree there is no support for a multi-queue in this PMD: reta >> size returned by rte_eth_dev_info_get() call is 0, while max_rx_queues and >> max_tx_queues are both 4. >> >> Linux ixgbevf-2.15.3 driver on the other hand successfully initializes 2 >> RSS queues: for some reason it always limits the number of RSS queues by 2. >> >> ixgbevf_main.c: line 2539 >> u16 rss = min_t(u16, num_online_cpus(), 2); >> >> The above is strange since if MRQE is set to 1010b there are 4 RSS queues >> available which seems to be the case in my AWS Guest. >> >> However, let's get back to DPDK. As I've mentioned above the SR-IOV >> function i have is RSS capable (to be 100% sure I've verified both queues >> are receiving packets in a multi-socket TCP test). And it's a shame I can't >> utilize it with a DPDK. > Yes, it is not yet supported. > But a patch was recently sent: > http://dpdk.org/ml/archives/dev/2014-December/010028.html Applying this patchset seems to break the NIC fast path functionality of a AWS Guest NIC. I'm still debugging it - will update u as soon as I have more specific info. vlad > >> I wonder if there are any blockers to add this capability to the ixgbevf >> PMD and if not is it scheduled to some time soon? > Please jump in the above thread for your questions. > > My turn to ask a question: > Does it mean DPDK is going to be supported in your system (OSv)? > I've seen it in the OSv roadmap. > Could we have more details about how it would be integrated? > > Thanks
[dpdk-dev] DPDK RSS support for ixgbevf PMD
Hi , > -Original Message- > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Vlad Zolotarov > Sent: Tuesday, December 16, 2014 11:36 PM > To: Thomas Monjalon > Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov > Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD > > > On 12/15/14 22:33, Thomas Monjalon wrote: > > 2014-12-15 21:11, Vladislav Zolotarov: > >> Hi, > >> I'm running an ixgbevf PMD on an AWS guests with extended networking > >> (SR-IOV functions of 82599 Intel's NIC) and noticed that even in the > >> current git tree there is no support for a multi-queue in this PMD: > >> reta size returned by rte_eth_dev_info_get() call is 0, while > >> max_rx_queues and max_tx_queues are both 4. > >> > >> Linux ixgbevf-2.15.3 driver on the other hand successfully > >> initializes 2 RSS queues: for some reason it always limits the number of > RSS queues by 2. > >> > >> ixgbevf_main.c: line 2539 > >> u16 rss = min_t(u16, num_online_cpus(), 2); > >> > >> The above is strange since if MRQE is set to 1010b there are 4 RSS > >> queues available which seems to be the case in my AWS Guest. > >> > >> However, let's get back to DPDK. As I've mentioned above the SR-IOV > >> function i have is RSS capable (to be 100% sure I've verified both > >> queues are receiving packets in a multi-socket TCP test). And it's a > >> shame I can't utilize it with a DPDK. > > Yes, it is not yet supported. > > But a patch was recently sent: > > http://dpdk.org/ml/archives/dev/2014-December/010028.html > > Applying this patchset seems to break the NIC fast path functionality of a > AWS Guest NIC. > I'm still debugging it - will update u as soon as I have more specific info. > Pls make sure enabling and using 4 queues on guest if you enable vf rss on host and if each vf has 4 queues, The packet could be received and put in any of 4 queues on guest depend on hash value of the packet. You could let packet IP address increased continuously and make those packets distribute to different queues. Thanks Changchun
[dpdk-dev] DPDK RSS support for ixgbevf PMD
On 12/17/14 03:03, Ouyang, Changchun wrote: > Hi , > >> -Original Message- >> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Vlad Zolotarov >> Sent: Tuesday, December 16, 2014 11:36 PM >> To: Thomas Monjalon >> Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov >> Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD >> >> >> On 12/15/14 22:33, Thomas Monjalon wrote: >>> 2014-12-15 21:11, Vladislav Zolotarov: >>>> Hi, >>>> I'm running an ixgbevf PMD on an AWS guests with extended networking >>>> (SR-IOV functions of 82599 Intel's NIC) and noticed that even in the >>>> current git tree there is no support for a multi-queue in this PMD: >>>> reta size returned by rte_eth_dev_info_get() call is 0, while >>>> max_rx_queues and max_tx_queues are both 4. >>>> >>>> Linux ixgbevf-2.15.3 driver on the other hand successfully >>>> initializes 2 RSS queues: for some reason it always limits the number of >> RSS queues by 2. >>>> ixgbevf_main.c: line 2539 >>>> u16 rss = min_t(u16, num_online_cpus(), 2); >>>> >>>> The above is strange since if MRQE is set to 1010b there are 4 RSS >>>> queues available which seems to be the case in my AWS Guest. >>>> >>>> However, let's get back to DPDK. As I've mentioned above the SR-IOV >>>> function i have is RSS capable (to be 100% sure I've verified both >>>> queues are receiving packets in a multi-socket TCP test). And it's a >>>> shame I can't utilize it with a DPDK. >>> Yes, it is not yet supported. >>> But a patch was recently sent: >>> http://dpdk.org/ml/archives/dev/2014-December/010028.html >> Applying this patchset seems to break the NIC fast path functionality of a >> AWS Guest NIC. >> I'm still debugging it - will update u as soon as I have more specific info. Hi, thanks for tips but I have a question below. > Pls make sure enabling and using 4 queues on guest Why should I enable all 4? Won't any number below or equal 4 work if (!) 4 queues are available? Note that there is a 82599 mode when only 2 RSS queues are available per VF: MRQE=1011b. > if you enable vf rss on host and if each vf has 4 queues, > The packet could be received and put in any of 4 queues on guest depend on > hash value of the packet. > You could let packet IP address increased continuously and make those packets > distribute to different queues. > > Thanks > Changchun
[dpdk-dev] DPDK RSS support for ixgbevf PMD
Hi > -Original Message- > From: Vlad Zolotarov [mailto:vladz at cloudius-systems.com] > Sent: Wednesday, December 17, 2014 4:47 PM > To: Ouyang, Changchun; Thomas Monjalon > Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov > Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD > > > On 12/17/14 03:03, Ouyang, Changchun wrote: > > Hi , > > > >> -Original Message- > >> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Vlad Zolotarov > >> Sent: Tuesday, December 16, 2014 11:36 PM > >> To: Thomas Monjalon > >> Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov > >> Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD > >> > >> > >> On 12/15/14 22:33, Thomas Monjalon wrote: > >>> 2014-12-15 21:11, Vladislav Zolotarov: > >>>> Hi, > >>>> I'm running an ixgbevf PMD on an AWS guests with extended > >>>> networking (SR-IOV functions of 82599 Intel's NIC) and noticed that > >>>> even in the current git tree there is no support for a multi-queue in > >>>> this > PMD: > >>>> reta size returned by rte_eth_dev_info_get() call is 0, while > >>>> max_rx_queues and max_tx_queues are both 4. > >>>> > >>>> Linux ixgbevf-2.15.3 driver on the other hand successfully > >>>> initializes 2 RSS queues: for some reason it always limits the > >>>> number of > >> RSS queues by 2. > >>>> ixgbevf_main.c: line 2539 > >>>> u16 rss = min_t(u16, num_online_cpus(), 2); > >>>> > >>>> The above is strange since if MRQE is set to 1010b there are 4 RSS > >>>> queues available which seems to be the case in my AWS Guest. > >>>> > >>>> However, let's get back to DPDK. As I've mentioned above the SR-IOV > >>>> function i have is RSS capable (to be 100% sure I've verified both > >>>> queues are receiving packets in a multi-socket TCP test). And it's > >>>> a shame I can't utilize it with a DPDK. > >>> Yes, it is not yet supported. > >>> But a patch was recently sent: > >>> http://dpdk.org/ml/archives/dev/2014-December/010028.html > >> Applying this patchset seems to break the NIC fast path functionality > >> of a AWS Guest NIC. > >> I'm still debugging it - will update u as soon as I have more specific > >> info. > > Hi, thanks for tips but I have a question below. > > > Pls make sure enabling and using 4 queues on guest > > Why should I enable all 4? Won't any number below or equal 4 work if (!) > 4 queues are available? > Note that there is a 82599 mode when only 2 RSS queues are available per > VF: MRQE=1011b. > Yes you are right, it could be 2 queues, the actual queue number per VF depends on the number of VF: VF number from 1~32: 4 queues per VF; VF number from 33~64: 2 queues per VF; The queue number is determined by the vf number at the pf init stage, If pf setup 4 queues for each vf and distribute packets to 4 queues, but vf only poll 1 or 2 queue, Then packets in other queues will be lost. You can try to poll 4 queues firstly to debug your issue. Thanks Changchun
[dpdk-dev] DPDK RSS support for ixgbevf PMD
On 12/18/14 03:32, Ouyang, Changchun wrote: > Hi > >> -Original Message- >> From: Vlad Zolotarov [mailto:vladz at cloudius-systems.com] >> Sent: Wednesday, December 17, 2014 4:47 PM >> To: Ouyang, Changchun; Thomas Monjalon >> Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov >> Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD >> >> >> On 12/17/14 03:03, Ouyang, Changchun wrote: >>> Hi , >>> >>>> -Original Message- >>>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Vlad Zolotarov >>>> Sent: Tuesday, December 16, 2014 11:36 PM >>>> To: Thomas Monjalon >>>> Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov >>>> Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD >>>> >>>> >>>> On 12/15/14 22:33, Thomas Monjalon wrote: >>>>> 2014-12-15 21:11, Vladislav Zolotarov: >>>>>> Hi, >>>>>> I'm running an ixgbevf PMD on an AWS guests with extended >>>>>> networking (SR-IOV functions of 82599 Intel's NIC) and noticed that >>>>>> even in the current git tree there is no support for a multi-queue in >>>>>> this >> PMD: >>>>>> reta size returned by rte_eth_dev_info_get() call is 0, while >>>>>> max_rx_queues and max_tx_queues are both 4. >>>>>> >>>>>> Linux ixgbevf-2.15.3 driver on the other hand successfully >>>>>> initializes 2 RSS queues: for some reason it always limits the >>>>>> number of >>>> RSS queues by 2. >>>>>> ixgbevf_main.c: line 2539 >>>>>> u16 rss = min_t(u16, num_online_cpus(), 2); >>>>>> >>>>>> The above is strange since if MRQE is set to 1010b there are 4 RSS >>>>>> queues available which seems to be the case in my AWS Guest. >>>>>> >>>>>> However, let's get back to DPDK. As I've mentioned above the SR-IOV >>>>>> function i have is RSS capable (to be 100% sure I've verified both >>>>>> queues are receiving packets in a multi-socket TCP test). And it's >>>>>> a shame I can't utilize it with a DPDK. >>>>> Yes, it is not yet supported. >>>>> But a patch was recently sent: >>>>> http://dpdk.org/ml/archives/dev/2014-December/010028.html >>>> Applying this patchset seems to break the NIC fast path functionality >>>> of a AWS Guest NIC. >>>> I'm still debugging it - will update u as soon as I have more specific >>>> info. >> Hi, thanks for tips but I have a question below. >> >>> Pls make sure enabling and using 4 queues on guest >> Why should I enable all 4? Won't any number below or equal 4 work if (!) >> 4 queues are available? >> Note that there is a 82599 mode when only 2 RSS queues are available per >> VF: MRQE=1011b. >> > Yes you are right, it could be 2 queues, the actual queue number per VF > depends on the number of VF: > VF number from 1~32: 4 queues per VF; > VF number from 33~64: 2 queues per VF; > The queue number is determined by the vf number at the pf init stage, > If pf setup 4 queues for each vf and distribute packets to 4 queues, but vf > only poll 1 or 2 queue, > Then packets in other queues will be lost. > You can try to poll 4 queues firstly to debug your issue. Sure. When I hardcoded the queues number to 4 all works like a charm. The problem is that we can't leave it this way. I've posted some comments to your patches (0 and 5). Since I've joined the list after the patches have been sent I had to paste the patches bodies into my email and they will most like not appear in the thread of the original series. I hope this won't be a big problem... ;) vlad > Thanks > Changchun >
[dpdk-dev] DPDK RSS support for ixgbevf PMD
Hi , > -Original Message- > From: Vlad Zolotarov [mailto:vladz at cloudius-systems.com] > Sent: Thursday, December 18, 2014 6:34 PM > To: Ouyang, Changchun; Thomas Monjalon > Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov > Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD > > > On 12/18/14 03:32, Ouyang, Changchun wrote: > > Hi > > > >> -Original Message- > >> From: Vlad Zolotarov [mailto:vladz at cloudius-systems.com] > >> Sent: Wednesday, December 17, 2014 4:47 PM > >> To: Ouyang, Changchun; Thomas Monjalon > >> Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov > >> Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD > >> > >> > >> On 12/17/14 03:03, Ouyang, Changchun wrote: > >>> Hi , > >>> > >>>> -Original Message- > >>>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Vlad > Zolotarov > >>>> Sent: Tuesday, December 16, 2014 11:36 PM > >>>> To: Thomas Monjalon > >>>> Cc: dev at dpdk.org; Avi Kivity; Gleb Natapov > >>>> Subject: Re: [dpdk-dev] DPDK RSS support for ixgbevf PMD > >>>> > >>>> > >>>> On 12/15/14 22:33, Thomas Monjalon wrote: > >>>>> 2014-12-15 21:11, Vladislav Zolotarov: > >>>>>> Hi, > >>>>>> I'm running an ixgbevf PMD on an AWS guests with extended > >>>>>> networking (SR-IOV functions of 82599 Intel's NIC) and noticed > >>>>>> that even in the current git tree there is no support for a > >>>>>> multi-queue in this > >> PMD: > >>>>>> reta size returned by rte_eth_dev_info_get() call is 0, while > >>>>>> max_rx_queues and max_tx_queues are both 4. > >>>>>> > >>>>>> Linux ixgbevf-2.15.3 driver on the other hand successfully > >>>>>> initializes 2 RSS queues: for some reason it always limits the > >>>>>> number of > >>>> RSS queues by 2. > >>>>>> ixgbevf_main.c: line 2539 > >>>>>> u16 rss = min_t(u16, num_online_cpus(), 2); > >>>>>> > >>>>>> The above is strange since if MRQE is set to 1010b there are 4 > >>>>>> RSS queues available which seems to be the case in my AWS Guest. > >>>>>> > >>>>>> However, let's get back to DPDK. As I've mentioned above the > >>>>>> SR-IOV function i have is RSS capable (to be 100% sure I've > >>>>>> verified both queues are receiving packets in a multi-socket TCP > >>>>>> test). And it's a shame I can't utilize it with a DPDK. > >>>>> Yes, it is not yet supported. > >>>>> But a patch was recently sent: > >>>>> http://dpdk.org/ml/archives/dev/2014-December/010028.html > >>>> Applying this patchset seems to break the NIC fast path > >>>> functionality of a AWS Guest NIC. > >>>> I'm still debugging it - will update u as soon as I have more specific > >>>> info. > >> Hi, thanks for tips but I have a question below. > >> > >>> Pls make sure enabling and using 4 queues on guest > >> Why should I enable all 4? Won't any number below or equal 4 work if > >> (!) > >> 4 queues are available? > >> Note that there is a 82599 mode when only 2 RSS queues are available > >> per > >> VF: MRQE=1011b. > >> > > Yes you are right, it could be 2 queues, the actual queue number per VF > depends on the number of VF: > > VF number from 1~32: 4 queues per VF; > > VF number from 33~64: 2 queues per VF; > > The queue number is determined by the vf number at the pf init stage, > > If pf setup 4 queues for each vf and distribute packets to 4 queues, > > but vf only poll 1 or 2 queue, Then packets in other queues will be lost. > > You can try to poll 4 queues firstly to debug your issue. > > Sure. When I hardcoded the queues number to 4 all works like a charm. > The problem is that we can't leave it this way. I've posted some comments to > your patches (0 and 5). Since I've joined the list after the patches have been > sent I had to paste the patches bodies into my email and they will most like > not appear in the thread of the original series. > I hope this won't be a big problem... ;) > Yes, I have seen your comments there, and respond it already, let's use that thread to discuss from now on. Thanks Changchun
